--- title: 'Linux Network Administration and Troubleshooting' source: 'https://academia.sh/en/courses/linux-network-administration' language: en updated: '2026-08-17T04:41:50+00:00' license: 'CC BY-SA 4.0' lessons: 13 outcomes: - "Reads and changes a server's network configuration" - 'Narrows a connectivity problem layer by layer and counts the candidate causes each test eliminates' - 'Configures firewall rules and remote access without locking itself out' --- # Linux Network Administration and Troubleshooting Reading a single server's network configuration and narrowing a connectivity failure by counting the candidate causes each test eliminates. ## Network Configuration Interfaces and addresses, the routing table, name resolution configuration, and the inventory of listening ports; how many candidate causes each reading eliminates. ### Interfaces and Addresses Source: https://academia.sh/en/courses/linux-network-administration/interfaces-and-addresses The interface list gives three distinct answers against eight candidate causes; two answers alone eliminate seven candidates each, the third eliminates only two and leaves six remaining. The previous course ended with storage: the layer where a fault looks quietest. A filling inode table produced no warning, a filling snapshot left no trace behind it. But that quiet had one advantage — the fault was **local**. There was one machine to look at, and every layer of that machine was reachable from the same shell. That advantage does not exist on a network. A fault can be both silent and **on someone else's machine**; it can even be on a path that belongs to no one. So the course's measure changes. The System Administration course counted what a tool cannot see. This course counts **how many candidate causes a test eliminates**. Its rule is this: a test's number is not the answer it gives, but the candidates it eliminates; **a test whose elimination count is not written is not considered measured.** The first lesson starts with the lowest-level test, the interface and address list. #### The Mock Server and Eight Candidate Causes A single mock server is followed throughout the course. On it runs a unit named `data-ingest`, listening on a port; the server connects to the network through an interface, exits outward through a default gateway, and asks a resolver for names. Someone trying to connect to this server says "the connection is not working." This is where diagnosis starts, and the sentence says nothing: eight separate faults produce exactly this sentence. - **AY1** — There are eight candidate causes, and each breaks **a single field** of the healthy state: interface down, no address, wrong gateway, missing route, silent resolver, service not listening, firewall dropping, large packet dropping. No two faults are assumed to occur at once. - **AY2** — **The oracle is the real fault.** Because we chose the setup, we know which field is broken; the person performing the test does not. The measurement uses this asymmetry. - **AY3** — There are eight **tests**; each corresponds to a tool and returns **one answer**. The answer splits the candidate set: candidates giving the same answer stay together, those giving a different answer are eliminated. - **AY4** — All counts are exhaustive. The count is computed over eight candidates and eight tests exhaustively; there is no randomness. This course is **a single machine's configuration**. Layer models, addressing arithmetic, name resolution protocols, and routing protocols are the subject of the Computer Networks curriculum; they are not built here and are left to the lessons there. The question here is not how the protocol works, but **what this machine's configuration says** and how many candidates that statement eliminates. Container networking and namespaces are also outside this course; they are left to the Kernel Interfaces and Isolation course. #### What the Interface List Shows An interface is the name the kernel gives to a piece of network hardware or a virtual link. The listing tool states two separate things for every interface: **administrative state** (has the interface been brought up) and **carrier state** (is a signal coming from the other end). These two are separate, and their separateness is the basis of the measurement. ```text # example dump — not executed $ ip link show 1: lo: mtu 65536 state UNKNOWN 2: eth0: mtu 1500 state UP 3: eth1: mtu 1500 state DOWN ``` The address is a separate question. An interface can be up, have carrier, and still carry no address at all; in that case no packet can be produced. The short-form address dump places the two questions side by side. ```text # example dump — not executed $ ip -brief addr show lo UNKNOWN 127.0.0.1/8 eth0 UP 203.0.113.24/24 eth1 DOWN ``` These two blocks are **not executed**, and no numeric claim is drawn from them. What the dump teaches is the format: the state field, the interface name, and the address field. Every address in this lesson is fictional; interface names also vary by the distribution's naming scheme, and that is said once here and not repeated. There are two ways to assign an address, and the difference between them is a frequent point of confusion in diagnosis. An address given with a command **belongs to runtime**: it is written into the kernel's table and lost on reboot. An address written into a configuration file or a network manager is **persistent**: it is reapplied at boot. The two can conflict. A server running with a manually assigned address can look fine for weeks and come up addressless at the first reboot. This is the same pattern as the persistent mount entry from the storage course: **running state and persistent configuration are two separate facts**, and the job is not done if only one has been tested. #### What the List Cannot See The interface list merges three separate questions into a single dump, and by merging them it loses some distinctions. Administrative state says whether the interface has been brought up, and this is a **configuration** decision. Carrier state says whether a signal is coming from the other end, and this is a **physical** fact. The third is whether the address exists, and that too is configuration. An interface that is up but has no carrier produces a line that looks up in the dump; even with the cable pulled out, the line stays in place and only the carrier field changes. If the person starting the diagnosis does not look at that field, they count the interface as healthy. The same merging exists on the address side. An interface can carry more than one address, and the dump lists them all one under another; which one will be used as the source on outgoing packets cannot be read from the list — routing decides that. Two interfaces getting an address on the same subnet also looks completely normal at the list level and can carry traffic to an unexpected interface. - **AY5** — The shared reference collapses these distinctions into **a single field**: the `interface-down` candidate means both that the interface is down and that the address is absent, because an address is not active on a down interface. The difference between carrier state and administrative state is not kept as a separate field in the model; this is a deliberate limit of the model, and it is exactly this limit that the sweep tests. #### A Test's Measure Is the Candidates It Eliminates The definition below is the shared reference used throughout the course: the mock server's healthy state, eight faults, eight tests. Each test returns the answer a tool would give. - **AY6** — Tests are invoked independently of one another; one test's answer does not change another's. What is measured is how many candidates each test eliminates **on its own**. - **AY7** — "Worst-case remaining" is the size of the **largest group** among the answers a test can give; it is a measure independent of the oracle and gives the test's ceiling. ```python """Shared reference: the mock server's healthy state, eight candidate faults, eight tests. What is measured is how many candidate causes a test ELIMINATES on its own.""" HEALTHY = {"interface_up": True, "address_present": True, "gateway_correct": True, "route_present": True, "resolver_responds": True, "service_listening": True, "firewall_allows": True, "path_intact": True} FAULT = { "interface-down": {"interface_up": False, "address_present": False}, "no-address": {"address_present": False}, "wrong-gateway": {"gateway_correct": False}, "missing-route": {"route_present": False}, "resolver-silent": {"resolver_responds": False}, "service-not-listening": {"service_listening": False}, "firewall-dropping": {"firewall_allows": False}, "large-packet-lost": {"path_intact": False}, } def state(fault, table): d = dict(HEALTHY) d.update(table[fault]) return d def test_interface(d): return ("up-addressed" if d["interface_up"] and d["address_present"] else "up-unaddressed" if d["interface_up"] else "down") def test_route(d): if not d["address_present"]: return "table-empty" return "complete" if d["route_present"] else "default-missing" def test_gateway_echo(d): return "reply" if d["interface_up"] and d["address_present"] else "no-reply" def test_remote_echo(d): return "reply" if (d["interface_up"] and d["address_present"] and d["gateway_correct"] and d["route_present"]) else "no-reply" def test_name_resolution(d): return "resolved" if d["resolver_responds"] else "unresolved" def test_listening(d): return "listening" if d["service_listening"] else "not-listening" def test_connection(d): if not (d["interface_up"] and d["address_present"] and d["gateway_correct"] and d["route_present"]): return "unreachable" if not d["firewall_allows"]: return "timeout" return "refused" if not d["service_listening"] else "established" def test_large_packet(d): return "passes" if (d["interface_up"] and d["address_present"] and d["gateway_correct"] and d["route_present"] and d["path_intact"]) else "blocked" TEST = {"interface": test_interface, "route": test_route, "gateway-echo": test_gateway_echo, "remote-echo": test_remote_echo, "name-resolution": test_name_resolution, "listening": test_listening, "connection": test_connection, "large-packet": test_large_packet} def groups(test, table=FAULT): o = {} for a in table: o.setdefault(TEST[test](state(a, table)), []).append(a) return o print("test distinct answers worst-case remaining") for s in TEST: o = groups(s) print(f" {s:14s} {len(o):9d} {max(len(v) for v in o.values()):13d}") print() print("interface test: candidates per answer") for answer, group in groups("interface").items(): print(f" {answer:14s} remaining {len(group):2d} eliminated {len(FAULT) - len(group):2d}" f" {', '.join(group)}") print() print("candidate set sweep") WITH_EXTRA = dict(FAULT, **{"no-cable": {"interface_up": False, "address_present": False}}) WITHOUT = {k: v for k, v in FAULT.items() if k != "no-address"} for ad, table in (("base 8 candidates", FAULT), ("+no-cable 9", WITH_EXTRA), ("-no-address 7", WITHOUT)): o = groups("interface", table) worst = max(len(v) for v in o.values()) down = len(o.get("down", [])) print(f" {ad:20s} distinct answers {len(o)} worst-case remaining {worst}" f" remaining in 'down' answer {down}" f" eliminated {len(table) - down}") ``` ``` test distinct answers worst-case remaining interface 3 6 route 3 5 gateway-echo 2 6 remote-echo 2 4 name-resolution 2 7 listening 2 7 connection 4 4 large-packet 2 5 interface test: candidates per answer down remaining 1 eliminated 7 interface-down up-unaddressed remaining 1 eliminated 7 no-address up-addressed remaining 6 eliminated 2 wrong-gateway, missing-route, resolver-silent, service-not-listening, firewall-dropping, large-packet-lost candidate set sweep base 8 candidates distinct answers 3 worst-case remaining 6 remaining in 'down' answer 1 eliminated 7 +no-cable 9 distinct answers 3 worst-case remaining 6 remaining in 'down' answer 2 eliminated 7 -no-address 7 distinct answers 2 worst-case remaining 6 remaining in 'down' answer 1 eliminated 6 ``` Three numbers stand side by side. **Oracle:** one of eight candidates is the real fault, and the side that built the setup knows which. **Test:** the interface list produces **three distinct answers** against these eight states — down, up-unaddressed, up-addressed. **Candidates eliminated:** seven, seven, and **two** depending on the answer; in the worst case, **six candidates** remain. #### The Three Answers Are Not Equal In the first table, the interface test giving three distinct answers makes it look like one of the more detailed tests; only the connection attempt gives more, four answers. But the number of distinct answers is not elimination power. The second table opens this up: two of the three answers point to a single candidate, the third leaves six candidates together. The practical reading of this is as follows. The interface test is **decisive when it finds the fault, and almost silent when it does not**. If the list says "down," the job is done: seven candidates are eliminated, one remains. If the list says "up-unaddressed," the job is done again. But if the list says, as expected, "up and addressed" — which is the most common answer the person starting the diagnosis encounters — only two of eight candidates are eliminated and six remain. The test's **expected** result is the result carrying the least information. This asymmetry gives the course's first general rule: **a test that gives a positive answer does not make a diagnosis, it only closes one door.** Looking at the interface list and saying "there's no problem here" is correct, and it leaves three-quarters of the candidates standing. Looking at the same list, on the other hand, and saying "the interface is up, so the network is working" is drawing six candidates' innocence from the elimination of two. The reason the test is still done first is not its elimination power but its **cost**. The interface list is read locally, has no side effect, returns in under a second, and when it is wrong, points definitively to two candidates. Tests that look stronger in the first table — the connection attempt and the remote echo, both leaving **four** candidates in the worst case — require reaching the far end, an open target, and a wait; both say "unreachable" or "no reply" while the interface is down, and that answer fits **four** separate candidates at once. This is the logic of the ordering: **cheap and decisive first, expensive and discriminating later.** How many candidates are eliminated gives a test's value; the ratio of value to cost decides the order it is done in. #### Candidate Set Sweep Because there is no randomness in this course, the measurement cannot be repeated with a second seed. The way to see how much the result depends on the setup is to **sweep the candidate set**: a candidate is added to the list and one is removed, and how the elimination counts change is read. - **AY8** — The added candidate, `no-cable`, is the case of the physical link being cut, and for the interface test it produces **the same answer** as `interface-down`. - **AY9** — The removed candidate, `no-address`, is the case of the interface being left without an address, and it alone fills the test's third answer. - **AY10** — The sweep changes only the candidate list; the healthy state, the test definitions, and the measurement method stay the same. The third table shows two separate fragilities. When a candidate is added, the "worst-case remaining" count stays fixed at six, meaning the test's ceiling does not change; but **the answer thought to be decisive loses its decisiveness**. In the nine-candidate list, the "down" answer still eliminates seven candidates, but what remains is no longer one candidate but **two**: `interface-down` and `no-cable` cannot be told apart from the interface list. What separates them is a different reading — carrier state. The test's power is not in the test, it is **in the candidate list**. When a candidate is removed, the answer count drops from three to **two**, and the "down" answer's elimination count drops from seven to **six**. The number shrank, but the test did not get weaker; what shrank was the problem. This shows a trap that will come up repeatedly in interpreting this measure: **the elimination count is not a property of the test, it is a joint property of the test and the candidate set.** Elimination counts measured in two setups are comparable only if the candidate lists are the same. The sweep's practical counterpart is writing down the candidate list before starting diagnosis. Without a written list, the elimination count cannot be computed and the order tests are done in is left to intuition; with a written list, every answer shows how many lines it strikes out. The rest of this course uses the same eight lines and, in each lesson, counts how the list narrows as one or two tests are added. The list itself is also a configuration decision: which faults are considered plausible directly determines which tests are found worth doing. #### Summary - This course reads a single machine's network configuration and measures every test by **how many candidate causes it eliminates**; a test whose elimination count is not written is not considered measured. - The mock server has eight candidate causes and eight tests; each fault breaks a single field of the healthy state, and the oracle is the real fault known to the side that built the setup. - The interface list gives **three distinct answers**, and the answers are not equal: the down and up-unaddressed answers each eliminate **seven** candidates, the expected up-addressed answer eliminates only **two** and leaves **six candidates** remaining. - The address at runtime and the address in persistent configuration are two separate facts; if only one has been tested, the configuration has not been verified. - The candidate set sweep shows the fragility of the elimination count: when a ninth candidate is added, the "down" answer no longer leaves a single candidate; when `no-address` is removed, the answer count drops from three to two. The elimination count is a joint property of the test and the candidate set, not of the test alone. #### Next Step Once the interface comes up as up and addressed, six candidates remain, and two of them concern where the packet is sent: the default gateway can be wrong, or the route may not exist at all. The next lesson reads the routing table and applies the same measure to it. What will be seen there is that the table gives three distinct answers, but it is unexpectedly hard-pressed to tell two candidates apart — a wrong gateway and a missing route: when the two echo-based tests are done back to back, these two candidates still sit in the same group. ### The Routing Table Source: https://academia.sh/en/courses/linux-network-administration/routing-table The routing table gives three answers against eight candidate causes and leaves five in the worst case; a wrong gateway and a missing route still sit in the same group after two echo tests. When the interface list gave the expected answer — up and addressed — only two of eight candidates were eliminated, and six remained. Two of these six concern where the packet **is sent**: the default gateway can be wrong, or the route to the destination may not exist at all. Both sit behind an up interface that has an address, and both are invisible from the interface list. This lesson reads the routing table and applies the same measure to it. The result is instructive in two respects. Reading the table leaves a narrower group than the interface list did — a stronger test. Against this, the first approach that comes to mind for telling these two apart — sending an echo — cannot separate them despite two separate tests. #### What the Table Says The routing table is made of rows that determine, for a destination address, which interface and which next hop a packet is sent through. Rows are not tried in order; every row has a destination prefix, and the kernel picks the **longest matching prefix**. The default gateway is the row with the shortest prefix under this rule, and for that reason it is used only when no other row matches. When an address is assigned to an interface, that subnet's row enters the table on its own; this is called a connected route, and it is what gives access to the local network. ```text # example dump — not executed $ ip route show default via 203.0.113.1 dev eth0 203.0.113.0/24 dev eth0 proto kernel scope link src 203.0.113.24 198.51.100.0/24 via 203.0.113.9 dev eth0 ``` It is also possible to ask for the decision for a single destination instead of reading the whole table. The decision query states three things at once: the chosen row, the outgoing interface, and which address will be written as the packet's source address. Source address selection does not show up in the table, and when it is chosen wrongly the fault becomes one-directional: the packet leaves, the reply cannot come back. ```text # example dump — not executed $ ip route get 198.51.100.42 198.51.100.42 via 203.0.113.9 dev eth0 src 203.0.113.24 ``` These two blocks are not executed, and no numeric claim is drawn from them; what they teach is the shape of the row. The addresses are fictional. How routing protocols work, how tables are distributed among neighbors, and how metrics are computed is the subject of the Computer Networks curriculum; in this lesson the table is taken as **given** and is only read. - **AY11** — The mock server's table consists of three rows: the default gateway, the local subnet's connected route, and a single static route written to a remote subnet. More than one routing table at a time is not assumed. - **AY12** — The `wrong-gateway` candidate is the case where the default row exists but the next hop it points to does not forward the packet. The `missing-route` candidate is the case where no row matches the destination at all. In the table, one is **present and wrong**, the other is **absent**. #### The Route Test's Measure ```python """Routing table test: how many of 8 candidate causes are eliminated. Same eight faults from the shared reference; this lesson reads the route and two echo tests.""" HEALTHY = {"interface_up": True, "address_present": True, "gateway_correct": True, "route_present": True, "resolver_responds": True, "service_listening": True, "firewall_allows": True, "path_intact": True} FAULT = { "interface-down": {"interface_up": False, "address_present": False}, "no-address": {"address_present": False}, "wrong-gateway": {"gateway_correct": False}, "missing-route": {"route_present": False}, "resolver-silent": {"resolver_responds": False}, "service-not-listening": {"service_listening": False}, "firewall-dropping": {"firewall_allows": False}, "large-packet-lost": {"path_intact": False}, } def state(fault, table): d = dict(HEALTHY) d.update(table[fault]) return d def test_route(d): if not d["address_present"]: return "table-empty" return "complete" if d["route_present"] else "default-missing" def test_gateway_echo(d): return "reply" if d["interface_up"] and d["address_present"] else "no-reply" def test_remote_echo(d): return "reply" if (d["interface_up"] and d["address_present"] and d["gateway_correct"] and d["route_present"]) else "no-reply" TEST = {"route": test_route, "gateway-echo": test_gateway_echo, "remote-echo": test_remote_echo} def group_by(tests, table=FAULT): o = {} for a in table: key = tuple(TEST[s](state(a, table)) for s in tests) o.setdefault(key, []).append(a) return o print("route test: candidates per answer") for answer, group in group_by(["route"]).items(): print(f" {answer[0]:14s} remaining {len(group):2d} eliminated {len(FAULT) - len(group):2d}" f" {', '.join(group)}") print() print("indistinguishable groups as the tool set grows") for tools in (["gateway-echo"], ["gateway-echo", "remote-echo"], ["gateway-echo", "remote-echo", "route"]): indistinguishable = [sorted(v) for v in group_by(tools).values() if len(v) > 1] print(f" {len(tools)} test {'+'.join(tools):38s} group {len(indistinguishable)}") for o in indistinguishable: print(f" {', '.join(o)}") print() print("candidate set sweep (route test)") WITH_EXTRA = dict(FAULT, **{"address-wrong-subnet": {"gateway_correct": False, "route_present": False}}) WITHOUT = {k: v for k, v in FAULT.items() if k != "wrong-gateway"} for ad, table in (("base 8", FAULT), ("+wrong-subnet 9", WITH_EXTRA), ("-wrong-gateway 7", WITHOUT)): o = group_by(["route"], table) missing = len(o.get(("default-missing",), [])) print(f" {ad:18s} distinct answers {len(o)} worst-case remaining " f"{max(len(v) for v in o.values())}" f" remaining in 'default-missing' answer {missing} eliminated {len(table) - missing}") ``` ``` route test: candidates per answer table-empty remaining 2 eliminated 6 interface-down, no-address complete remaining 5 eliminated 3 wrong-gateway, resolver-silent, service-not-listening, firewall-dropping, large-packet-lost default-missing remaining 1 eliminated 7 missing-route indistinguishable groups as the tool set grows 1 test gateway-echo group 2 interface-down, no-address firewall-dropping, large-packet-lost, missing-route, resolver-silent, service-not-listening, wrong-gateway 2 test gateway-echo+remote-echo group 3 interface-down, no-address missing-route, wrong-gateway firewall-dropping, large-packet-lost, resolver-silent, service-not-listening 3 test gateway-echo+remote-echo+route group 2 interface-down, no-address firewall-dropping, large-packet-lost, resolver-silent, service-not-listening candidate set sweep (route test) base 8 distinct answers 3 worst-case remaining 5 remaining in 'default-missing' answer 1 eliminated 7 +wrong-subnet 9 distinct answers 3 worst-case remaining 5 remaining in 'default-missing' answer 2 eliminated 7 -wrong-gateway 7 distinct answers 3 worst-case remaining 4 remaining in 'default-missing' answer 1 eliminated 6 ``` Three numbers side by side. **Oracle:** in this round the real fault is `wrong-gateway`; the default row sits in place and the node it points to does not forward the packet. **Test:** the routing table is read, and the answer is **complete** — one of three distinct answers. **Candidates eliminated:** only **three**; **five candidates** remain. The same test gives a completely different result with a different oracle. Had the real fault been `missing-route`, the answer would have been **default-missing**, **seven candidates** would be eliminated, and one candidate would remain. Had the answer been **table-empty**, six candidates would be eliminated. The asymmetry from the interface list exists here too, and in the same direction: the test is decisive at seeing an **absence** in the table, and almost blind at seeing a **wrongness** in the table. The reason is that the table is not a record of correctness but a record of **intent**. A row states where the packet is sent; it does not state whether it arrives. This distinction is directly useful when reading the table. Whether the default row **exists** is a verifiable fact and is answered definitively just by looking at the table. Whether the address in the row is **correct** is not a verifiable fact: whether that address is actually a router, whether it forwards the packet, and whether it has a path to the destination while forwarding, is outside the table's knowledge. In this respect the table resembles a file system entry: an entry existing does not show that where it points is correct. In the storage course, the persistent mount entry was correct on paper while mounting the wrong device; here too, the default row can be correct on paper while pointing to an address that forwards the packet nowhere. #### Why the Two Candidates Cannot Be Separated The second table shows why the `wrong-gateway` and `missing-route` pair is a hard pair. Sending an echo looks like the most natural way to separate these two candidates, and two separate echo tests can be done: one to the local gateway, the other to a remote address. - **AY13** — The gateway echo depends only on the interface being up and addressed; the local network is reached through the connected route, and the default row plays no part in this test. - **AY14** — The remote echo requires **all four** of the interface, the address, the gateway, and the route to be correct at once. This test is done directly by address, with name resolution not involved. - **AY15** — The echo itself is assumed not to be blocked; the case where the echo reply is filtered is not this topic's problem but the firewall lessons'. Looked at with a single echo test, eight candidates drop into **two groups**, and the large group carries six candidates. When the second echo is added, the group count rises to **three**: the test genuinely does work, splitting the six-candidate group in two. But it does not split where expected. `wrong-gateway` and `missing-route` **stay together in the new group**, because both produce the same pattern: the gateway echo replies, the remote echo does not. Two tests, two waiting periods, and two separate outputs were spent; this pair is **still not separated**. The reason for this result is not the tests' weakness but **the shape of the dependency chain**. For the remote echo to succeed, all four fields must be correct at once; when any one of the fields breaks, the answer is the same answer. The more fields a test depends on, the more candidates its negative answer fits at once. Tests do not get stronger as they approach the far end of the connection chain — quite the opposite, **their power to discriminate drops**; what they gain is coverage. The remote echo tests four fields at once in a single step, and for that reason it is a good tool for saying "everything is fine" and a poor one for saying "the problem is here." What does the separating is the third test, and that test sends no packet at all. When the table is read, the `missing-route` candidate gives the **default-missing** answer, the `wrong-gateway` candidate gives the **complete** answer; the group splits apart and the indistinguishable-group count drops from three to **two**. The course's second claim is paid off here for the first time: **more output does not mean better diagnosis.** Reading a local file does in a single step what two remote tests could not do. #### Source Address Selection and the One-Directional Fault The table has one more output that does not show. When a packet is produced, its source address is also chosen, and the choice is made not by the application but by routing: once the row to use is determined, one of the addresses bound to that row's outgoing interface is written as the source. The decision query's last field shows this choice; in the table dump it can only be read indirectly, from the connected route's source field. When this choice is wrong, the resulting fault pattern is distinctive. The packet reaches the destination, the destination produces a reply, and sends the reply to the address written as the source; if that address is not known on the return path, the reply is lost. What is seen on the local machine is "no reply," and this is the same picture produced by a wrong gateway. This pattern commonly arises on a server with two interfaces: outbound traffic leaves through one interface, return traffic is routed to the other, and no table row looks wrong. The fault's distinguishing signature is that two attempts to the same destination with two different source addresses give different results; a single attempt does not show this signature, because the absence of a reply is the same absence of a reply in both cases. - **AY16** — Source address selection is not kept as a separate field in the shared reference; on the single-interface mock server this choice carries no ambiguity. The multi-interface case is outside the model, and the pattern above is a caution about the reading, not about the model. #### Adding and Removing a Route Adding a route also has two forms, and the difference is the same as with address assignment: a route written to runtime is lost on reboot, a route written to configuration comes back at boot. When a route added by hand during diagnosis fixes the problem, the job is not considered done; the same row must also be carried into the persistent configuration. When two rows are written to the same destination, prefix length decides which one is used; if the prefixes are also equal, the metric given to the rows decides, and the smaller one is chosen. This is a frequent source of ambiguity in diagnosis: two default rows to the same destination can enter from two separate sources — a row given by hand and a row the address configuration added on its own — and the table dump shows both. Whoever looks at the dump counts both rows as correct; only one is actually used. The decision query removes this ambiguity, because it prints not the whole table but the **decision actually made**. Operations that change or delete the default row are a separate class. Changing the default gateway on a remotely managed machine can cut off the very session that wrote the command, and because the command does not return, no correction can be made either. This lesson does not give that command sequence **in a fully runnable form.** The preventive path has three steps, and all three will recur through this course: writing the change out first with a **dry run**, placing the change behind a **timer-based rollback** task, and providing management access through a separate channel **not affected** by the change. #### Candidate Set Sweep The third table shakes the candidate list in two directions. - **AY17** — The added candidate, `address-wrong-subnet`, is the case where the address is given from the wrong subnet; in that case the gateway becomes unreachable and the default row cannot be established, meaning two fields break at once. - **AY18** — The removed candidate, `wrong-gateway`, is the candidate the table reading stays weakest on. When the candidate is added, the distinct-answer count stays fixed at three, the worst-case remaining at five. What changes, again, is **the answer thought to be decisive**: the `default-missing` answer eliminates seven of nine candidates but leaves not one but **two** behind. In other words, the reading "there's no default row" is no longer a diagnosis but a narrowing down to two candidates. When the candidate is removed, the worst-case remaining drops from five to **four**, and the candidates the `default-missing` answer eliminates go from seven to **six**. The test is the same test; only the list changed. These two rows show why treating elimination counts as an absolute measure is wrong. The route reading being a test that "leaves five candidates in the worst case" is a number tied to the eight-candidate list. As the list grows, the test's ceiling rises; as it narrows, the ceiling falls; the one thing that stays fixed is **which field the test sees**. Ranking tests by this invariant property is a more durable habit than ranking them by elimination counts: the route reading always sees the route field, no matter what the candidate list is. #### Summary - The routing table is a record of intent: it states where the packet is sent, not whether it arrives there. Row selection follows the longest matching prefix rule, and the default gateway is the row with the shortest prefix. - The route test gives **three distinct answers** and leaves **five candidates** in the worst case: the `complete` answer eliminates only **three** candidates, the `default-missing` answer **seven**, the `table-empty` answer **six**. The test sees an absence in the table; it does not see a wrongness in the table. - The `wrong-gateway` and `missing-route` pair **cannot be separated by two echo tests**: both reply to the local echo and do not reply to the remote echo. The indistinguishable-group count is three after two tests, and two once the table reading is added. - The separation is done by a local reading that sends no packet at all; the two remote tests, which produce more output, cannot do it. - The sweep shows the elimination count depends on the candidate list: when a ninth candidate is added, the `default-missing` answer leaves two candidates; when `wrong-gateway` is removed, the worst-case remaining drops from five to four. #### Next Step Once the table reading gives the `complete` answer, four of the remaining five candidates concern not the packet's path anymore but **what is being asked for**: the destination's name may not resolve, the service may not be listening, the firewall may be dropping, or a large packet may be getting lost on the way. The next lesson looks at the first of these four and reads the resolver configuration. The number measured there is one of the course's smallest: the name resolution test, when it gives the expected answer, eliminates **only one** of eight candidates. Why such a weak test is still done is that lesson's real question. ### Name Resolution Source: https://academia.sh/en/courses/linux-network-administration/name-resolution The name resolution test, when it gives the expected answer, eliminates only one of eight candidates; when it fails, it eliminates seven, and this asymmetry explains why the course's weakest test is still done. When the routing table gave the expected answer — `complete` — five candidates remained. Four of these no longer concern the packet's path but **what is being asked for**: the destination's name may not resolve, the service may not be listening, the firewall may be dropping, or a large packet may be getting lost on the way. This lesson looks at the first of the four. The number to be measured is one of the course's smallest, and the lesson's real question is what that smallness means: if a test, when it gives the expected answer, eliminates **only one** of eight candidates, why is that test still done at all. #### What the Resolver Configures Name resolution is the conversion of a name into an address, and this job is not done in a single place. The query first hits the **name service order**: the system reads from a configuration file which source is tried in which order. The first source in the order is usually a local hosts file; after it comes a local cache or a stub resolver; last is the query sent to the resolver on the network. When a name is found in the local file, the network is never reached, and the answer the network resolver would have given is never seen. ```text # example dump — not executed $ cat /etc/resolv.conf nameserver 192.0.2.53 search example options timeout:2 attempts:2 ``` Three fields of the configuration determine three separate behaviors. The **resolver address** states where the query goes; when more than one is written, the first is tried, and the next is tried only when no reply comes. The **search domain** gives the suffix added to a name with no dot in it; a query made with a short name and the same query made with a fully qualified name can therefore produce different results. The **timeout and retry count** determine how long a fault takes to show: a two-second timeout with two attempts makes every query on a silent resolver wait four seconds, and at the application layer this shows up not as a name resolution fault but as **slowness**. ```text # example dump — not executed $ getent ahosts record-store.example 198.51.100.42 STREAM record-store.example 198.51.100.42 DGRAM 198.51.100.42 RAW ``` The choice of query tool also changes the result. Tools that ask the resolver directly **skip** the name service order and test only the network resolver; testing the path the system actually uses requires a tool that goes through the whole order. This distinction produces a concrete result in diagnosis: one query tool can resolve a name that the application cannot, or the other way around. How names are resolved over the network, which record types the query and the answer carry, and how the authoritative server hierarchy works is the subject of the Computer Networks curriculum; no protocol is built here, only **this machine's resolver configuration** is read. - **AY19** — On the mock server, the name service order consists of two sources: the local hosts file and a single network resolver. The name being looked up is assumed not to be present in the hosts file. - **AY20** — The `resolver-silent` candidate is the resolver never replying to the query at all. A resolver giving a wrong answer is a separate candidate and is not in the base list; it is added in the sweep. The path to the resolver itself being sound is accepted as a requirement of the single-fault assumption. #### The Order Itself Is a Source of Faults The name service order produces two separate traps in diagnosis, and both are common. The first is **the precedence of the local entry**. A line written into the hosts file wins regardless of what the network resolver says. When a line added temporarily during a migration is forgotten in place, the system keeps connecting to the old address for months, and because the record on the network has been corrected, no query tool shows the fault — a query tool that skips the order returns the correct address, while the application goes to the wrong one. The only reading that makes the distinction is a query that goes through the order. The second is **the silent expansion of the search domain**. When a name with no dot in it is queried, the suffix defined for the system is appended, and the query turns into a different name. The same short name can expand to two different fully qualified names on two machines, because their search domains differ. For this reason, carrying a test result from one machine to another is not reliable: the name tested is not the name written. Writing the fully qualified name in configuration files closes this ambiguity at the source. A third behavior is not a fault but hides one: when a source in the order does not reply, the query falls through to the next. The fall-through is silent and shows up only as **duration**. In a configuration with a two-second timeout and two attempts, a silent resolver adds four seconds to every query; in the application log, the counterpart of this is not an error but a delay. The pattern from the System Administration course applies here too: **the visible symptom and the real fault sit at different layers.** #### The Test's Measure: One Candidate ```python """Name resolution test: how many of the same eight candidate causes are eliminated, and how many candidates it SEPARATES when added to the previous lessons' test set.""" HEALTHY = {"interface_up": True, "address_present": True, "gateway_correct": True, "route_present": True, "resolver_responds": True, "service_listening": True, "firewall_allows": True, "path_intact": True} FAULT = { "interface-down": {"interface_up": False, "address_present": False}, "no-address": {"address_present": False}, "wrong-gateway": {"gateway_correct": False}, "missing-route": {"route_present": False}, "resolver-silent": {"resolver_responds": False}, "service-not-listening": {"service_listening": False}, "firewall-dropping": {"firewall_allows": False}, "large-packet-lost": {"path_intact": False}, } def state(fault, table): d = dict(HEALTHY) d.update(table[fault]) return d def test_route(d): if not d["address_present"]: return "table-empty" return "complete" if d["route_present"] else "default-missing" def test_gateway_echo(d): return "reply" if d["interface_up"] and d["address_present"] else "no-reply" def test_remote_echo(d): return "reply" if (d["interface_up"] and d["address_present"] and d["gateway_correct"] and d["route_present"]) else "no-reply" def test_name_resolution(d): return "resolved" if d["resolver_responds"] else "unresolved" TEST = {"route": test_route, "gateway-echo": test_gateway_echo, "remote-echo": test_remote_echo, "name-resolution": test_name_resolution} def group_by(tests, table=FAULT): o = {} for a in table: key = tuple(TEST[s](state(a, table)) for s in tests) o.setdefault(key, []).append(a) return o print("name resolution test: candidates per answer") for answer, group in group_by(["name-resolution"]).items(): print(f" {answer[0]:11s} remaining {len(group):2d} eliminated {len(FAULT) - len(group):2d}" f" {', '.join(group)}") print() PREVIOUS = ["gateway-echo", "remote-echo", "route"] print("when added to the previous lessons' test set") for tests in (PREVIOUS, PREVIOUS + ["name-resolution"]): indistinguishable = [sorted(v) for v in group_by(tests).values() if len(v) > 1] covered = sum(len(o) for o in indistinguishable) print(f" {len(tests)} tests group {len(indistinguishable)} indistinguishable candidates {covered}") for o in indistinguishable: print(f" {', '.join(o)}") print() print("candidate set sweep (name resolution test)") WITH_EXTRA = dict(FAULT, **{"resolver-wrong-record": {"resolver_responds": False}}) WITHOUT = {k: v for k, v in FAULT.items() if k != "resolver-silent"} for ad, table in (("base 8", FAULT), ("+wrong-record 9", WITH_EXTRA), ("-resolver-silent 7", WITHOUT)): o = group_by(["name-resolution"], table) row = " ".join(f"{y[0]}: eliminated {len(table) - len(k):2d} remaining {len(k):2d}" for y, k in sorted(o.items())) print(f" {ad:21s} distinct answers {len(o)} {row}") ``` ``` name resolution test: candidates per answer resolved remaining 7 eliminated 1 interface-down, no-address, wrong-gateway, missing-route, service-not-listening, firewall-dropping, large-packet-lost unresolved remaining 1 eliminated 7 resolver-silent when added to the previous lessons' test set 3 tests group 2 indistinguishable candidates 6 interface-down, no-address firewall-dropping, large-packet-lost, resolver-silent, service-not-listening 4 tests group 2 indistinguishable candidates 5 interface-down, no-address firewall-dropping, large-packet-lost, service-not-listening candidate set sweep (name resolution test) base 8 distinct answers 2 resolved: eliminated 1 remaining 7 unresolved: eliminated 7 remaining 1 +wrong-record 9 distinct answers 2 resolved: eliminated 2 remaining 7 unresolved: eliminated 7 remaining 2 -resolver-silent 7 distinct answers 1 resolved: eliminated 0 remaining 7 ``` Three numbers side by side. **Oracle:** in this round the real fault is not `resolver-silent`; some other field is broken on the mock server, and the resolver is working. **Test:** the name query is made, and its answer is `resolved`. **Candidates eliminated:** **one**; **seven candidates** remain. This is one of the lowest values among the eight tests the course measures; only the listening-socket list gives the same number. #### The Value of a Weak Test The name resolution test produces two answers, and the gap between the two answers is the largest in the course: `resolved` eliminates one candidate, `unresolved` eliminates seven. Same command, same duration, same effort; the information it carries differs by a factor of seven. The test's value is not in the test, it is **in which answer it gives.** The practical reading of this asymmetry has two parts. First: when name resolution fails, the diagnosis is over. Seven candidates are eliminated, one remains, and what needs fixing is clear. Second: when name resolution succeeds, it looks as if nothing was learned, yet something was — a **confounder** has been removed. Every test done by name asks two questions at once: does the name resolve, and is the target reachable. A connection attempt's failure, made without resolution being verified separately, does not show which of these two questions it belongs to. The name resolution test **fixes the meaning** of every test that follows it. The second table turns this contribution into a number. The previous lesson's test set — the local echo, the remote echo, and the table reading — left eight candidates unseparated across two groups, and those groups held **six candidates** in total. When name resolution is added, the group count stays at two, and the indistinguishable candidate count drops to **five**. So the test's contribution is exactly **one candidate**, and that one candidate is the one no other test can separate on its own. None of the four tests can separate the `resolver-silent` candidate without touching it, because when the resolver is silent, packets keep flowing along the path without any problem. This shows why ranking tests by elimination power alone is not enough. How many candidates a test eliminates on its own and how many candidates it **separates** when added to a set are separate measures, and the two do not move in the same direction. A strong test produces no contribution when it eliminates candidates the tests before it have already separated; a weak test becomes irreplaceable when it looks at the single field no other tool can see. Name resolution is of the second kind. When building a diagnostic order, the question to ask is not "how many candidates does this test eliminate" but "how many candidates **invisible to anything else** does this test separate." #### A Name Resolving Is Not Reachability When resolution succeeds, what comes back is an address, and not the information that the address is correct. The record may be stale, the target may have moved to another address, the answer may be coming from a cache. The cache plays a double role here. A positive answer is kept for the lifetime written into the record, and no change is seen until that time elapses; a negative answer can also be kept, and a corrected record appears later than expected. In both cases, what the test sees is not the network's current state but the state of **a moment in the past**. A name can also resolve to more than one address, and the returned list can carry more than one address family. In this case, which address to connect to is chosen not by the resolver but by whoever establishes the connection, and the selection order depends on the system's configuration. The resulting fault pattern is misleading in diagnosis: the name resolves correctly, the path to the first address in the list is broken, the path to the second works, and the result varies from attempt to attempt. If the same command run twice in a row gives two different results, the first place to look is the **list** resolution returns; a tool that shows a single-line answer hides this list. - **AY21** — There is no cache in the shared reference; the resolver either replies or does not. Lifetime and negative caching are outside the model, and the caution here is part of the reading, not of the measurement. - **AY22** — The name resolution test is assumed independent of the result of tests done by address; the combined case where the path to the resolver itself is broken is outside the single-fault assumption. The working habit that follows from this is one sentence: **test by address first, then by name.** If an attempt by address succeeds and the same attempt by name fails, the fault is in resolution and no other test is needed. If both fail, resolution stops being a candidate and the problem is in the layers below. When the order is reversed — trying by name first and seeing failure — the answer obtained covers two candidates at once and a second test is needed to separate them. The same two commands, in a different order, eliminate a different number of candidates. #### Candidate Set Sweep The third table shakes the candidate list in two directions again, and in this lesson one of the results is the sharpest in the course. - **AY23** — The added candidate, `resolver-wrong-record`, is the case where the resolver replies but returns the wrong address. In the model this is represented as resolution not giving the expected result, and it produces the same answer as `resolver-silent`. - **AY24** — The removed candidate, `resolver-silent`, is the candidate that alone fills the test's negative answer. When the candidate is added, the number of candidates the `resolved` answer eliminates rises from one to **two**; the test does twice the work, and it does so with nothing changed in its own definition. The `unresolved` answer still eliminates seven candidates, but it now leaves not one but **two** behind: a silent resolver and a resolver giving a wrong answer cannot be separated by this test. What does the separating is looking at the answer's **content**, not its presence. When the candidate is removed, the result is harsher. In the seven-candidate list, the number of answers the test can give drops to **one**, and the candidates it eliminates become **zero**. The test still runs, still produces an answer, still spends time — and eliminates no candidate. This is the sharpest demonstration of the course's rule: **the output a test produces and the information it carries are separate things.** A test with no counterpart in the candidate list, however rich its output, is not unmeasured — it is **worthless**. The diagnostic counterpart of this is a regular review of tests done out of habit: whichever candidate a test entered the list to separate, once that candidate drops from the list, the test should drop too. #### Summary - Name resolution is not done in a single place: the query goes through the name service order, and the local hosts file and cache can produce an answer without reaching the network; tools that ask the resolver directly skip this order and do not test the path the system actually uses. - The name resolution test gives **two answers**, and the gap between them is the largest in the course: `resolved` eliminates only **one** candidate and leaves **seven** remaining, `unresolved` eliminates **seven** and finishes the diagnosis. - The test's contribution when it gives the expected answer is removing a confounding variable: it separates exactly **one** of the six candidates the previous three tests could not separate, bringing the indistinguishable count from six to five. No other test can separate that candidate. - A name resolving does not mean the target is reachable; because of caching and lifetime, the returned answer can be the state of a moment in the past. The working rule is to test by address first, then by name. - The sweep gives two results: when a resolver giving a wrong answer is added, the `resolved` answer starts eliminating two candidates; when `resolver-silent` is removed, the test's answer count drops to one and the candidates it eliminates drop to **zero**. A test with no counterpart in the candidate list is worthless. #### Next Step Once name resolution is also confirmed, three candidates remain: the service may not be listening, the firewall may be dropping the packet, or a large packet may be getting lost on the way. All three look as if they leave a place to look on the server itself — the list of listening sockets. The next lesson pulls up that list and puts two numbers side by side: the local inventory eliminates **one** of eight candidates, a single connection attempt made from outside eliminates **seven**. Same fault, same moment, a sevenfold difference between two tests. That lesson's question is where this difference comes from: the listening list sees a single field, while the connection attempt separates four distinct outcomes from one another. ### Ports and Listening Services Source: https://academia.sh/en/courses/linux-network-administration/ports-and-listening-services The local listening list eliminates only one of eight candidate causes; a single connection attempt made from outside eliminates seven at once, and across eight faults the total is 14 against 42. Once name resolution was also confirmed, three candidates remained: the service may not be listening, the firewall may be dropping the packet, or a large packet may be getting lost on the way. All three look as if they leave a place to look on the server itself — the list of listening sockets. The list is local, fast, detailed, and looks directly at where the problem passes through. This lesson pulls up that list and places a second test next to it: a single connection attempt made from outside. The two tests are measured at the same time, against the same fault, and the gap between them is one of the largest in the course. #### What a Listening Port Is When a service starts listening on a port, a record is created in the kernel: which address-and-port pair will accept incoming requests. The local inventory lists these records. The list's most important column is not the port number, it is the **bind address**. ```text # example dump — not executed $ ss -ltn State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 0.0.0.0:9443 0.0.0.0:* LISTEN 0 128 127.0.0.1:7070 0.0.0.0:* ``` The two rows have the same shape and tell two different facts. The first accepts requests coming from every interface; the second accepts requests only from the loopback interface — that is, from the machine itself. Seen from outside, the second row's counterpart is a port that is **not listening at all**. The inventory writes "LISTEN" for both rows. The process column is a separate reading and requires privilege: a listing tool run without privilege shows which ports are open, not which process opened them. This costs a step in diagnosis — which port is open is known, which unit opened it is not. How port numbers are assigned, which ranges are reserved for ephemeral ports, and how the transport layer's connection-establishment behavior works is the subject of the Computer Networks curriculum; no protocol is built here, only **this machine's inventory** is read. - **AY25** — On the mock server, the `data-ingest` unit listens on every interface; a second, local management endpoint listens only on loopback and is never exposed outward. - **AY26** — In the shared reference, the listening list and the connection attempt depend on **the same field**: if the service is listening, the list shows it and the connection is established. A service bound only to loopback — appearing to listen in the list while being refused from outside — is **outside** the model, and the caution above is part of the reading, not of the measurement. #### Three Things That Get Confused When Reading the List The inventory gathers three separate concepts into a single table, and because all three are described with the word "listening," they get confused with one another. The first is **ports that are not connection-oriented**. In the connection-establishing transport form, a port explicitly enters a listening state and appears as such in the list; in the connectionless form there is no such state, only a port being bound is seen. The two kinds sit side by side in the same list and are read with the same word, yet for the second kind, "listening" means only "packets arriving at this port are delivered to a process." The second is **the separation of socket state from service health**. The existence of a listening record does not show that the process behind it is actually processing requests. A process that does not accept the request but leaves the socket open looks healthy in the list; the listen backlog fills up, and new connections start being dropped past a certain point. The measurement from the System Administration course is not repeated here; its result there was this: a unit looking "running" and a unit doing its job are two separate facts. Its network-side counterpart is the difference between a listening record existing and a request being answered. The third is **the difference between persistent configuration and running state**, and this is the pattern repeated since this topic's first lesson. A service started by hand shows up in the list; for it to start at boot, it must also be enabled separately. The inventory shows only the present moment. #### The Two Tests' Measure ```python """Elimination power of the local listening list versus an external connection attempt. Same eight candidate causes, same moment, two separate tests.""" HEALTHY = {"interface_up": True, "address_present": True, "gateway_correct": True, "route_present": True, "resolver_responds": True, "service_listening": True, "firewall_allows": True, "path_intact": True} FAULT = { "interface-down": {"interface_up": False, "address_present": False}, "no-address": {"address_present": False}, "wrong-gateway": {"gateway_correct": False}, "missing-route": {"route_present": False}, "resolver-silent": {"resolver_responds": False}, "service-not-listening": {"service_listening": False}, "firewall-dropping": {"firewall_allows": False}, "large-packet-lost": {"path_intact": False}, } def state(fault, table): d = dict(HEALTHY) d.update(table[fault]) return d def test_listening(d): return "listening" if d["service_listening"] else "not-listening" def test_connection(d): if not (d["interface_up"] and d["address_present"] and d["gateway_correct"] and d["route_present"]): return "unreachable" if not d["firewall_allows"]: return "timeout" return "refused" if not d["service_listening"] else "established" TEST = {"listening": test_listening, "connection": test_connection} def groups(test, table=FAULT): o = {} for a in table: o.setdefault(TEST[test](state(a, table)), []).append(a) return o for s in TEST: print(f"{s} test: candidates per answer") for answer, group in groups(s).items(): print(f" {answer:12s} remaining {len(group):2d} eliminated {len(FAULT) - len(group):2d}" f" {', '.join(group)}") print() print("eliminated candidates per oracle") print(f" {'real fault':27s} {'listening':>20s} {'connection':>20s}") total = {"listening": 0, "connection": 0} for real_fault in FAULT: row = "" for s in TEST: answer = TEST[s](state(real_fault, FAULT)) eliminated = len(FAULT) - len(groups(s)[answer]) total[s] += eliminated row += f" {answer:12s} {eliminated:2d} " print(f" {real_fault:27s} {row}") print(f" {'TOTAL eliminated':27s} {total['listening']:18d} {total['connection']:19d}") print() print("candidate set sweep") WITH_EXTRA = dict(FAULT, **{"service-crashed": {"service_listening": False}}) WITHOUT = {k: v for k, v in FAULT.items() if k != "firewall-dropping"} for ad, table in (("base 8", FAULT), ("+service-crashed 9", WITH_EXTRA), ("-firewall 7", WITHOUT)): d = groups("listening", table) b = groups("connection", table) dk = len(d.get("not-listening", [])) bk = len(b.get("refused", [])) print(f" {ad:18s} listening: answers {len(d)}" f" 'not-listening' eliminated {len(table) - dk} remaining {dk}" f" | connection: answers {len(b)} worst-case remaining " f"{max(len(v) for v in b.values())}" f" 'refused' remaining {bk}") ``` ``` listening test: candidates per answer listening remaining 7 eliminated 1 interface-down, no-address, wrong-gateway, missing-route, resolver-silent, firewall-dropping, large-packet-lost not-listening remaining 1 eliminated 7 service-not-listening connection test: candidates per answer unreachable remaining 4 eliminated 4 interface-down, no-address, wrong-gateway, missing-route established remaining 2 eliminated 6 resolver-silent, large-packet-lost refused remaining 1 eliminated 7 service-not-listening timeout remaining 1 eliminated 7 firewall-dropping eliminated candidates per oracle real fault listening connection interface-down listening 1 unreachable 4 no-address listening 1 unreachable 4 wrong-gateway listening 1 unreachable 4 missing-route listening 1 unreachable 4 resolver-silent listening 1 established 6 service-not-listening not-listening 7 refused 7 firewall-dropping listening 1 timeout 7 large-packet-lost listening 1 established 6 TOTAL eliminated 14 42 candidate set sweep base 8 listening: answers 2 'not-listening' eliminated 7 remaining 1 | connection: answers 4 worst-case remaining 4 'refused' remaining 1 +service-crashed 9 listening: answers 2 'not-listening' eliminated 7 remaining 2 | connection: answers 4 worst-case remaining 4 'refused' remaining 2 -firewall 7 listening: answers 2 'not-listening' eliminated 6 remaining 1 | connection: answers 3 worst-case remaining 4 'refused' remaining 1 ``` Three numbers side by side. **Oracle:** in this round the real fault is `firewall-dropping`; the service is listening, packets are arriving, and the filter silently drops them. **Test:** the local listening list is pulled up and its answer is `listening`; at the same time a connection is attempted from outside and its answer is `timeout`. **Candidates eliminated:** the listening list, **one**; the connection attempt, **seven**. The list leaves seven candidates, the connection attempt leaves one and finishes the diagnosis. #### The Power of Four Answers The third table shows this gap is not specific to a single fault. Summed across all eight faults, the listening list eliminates **14 candidates**, the connection attempt **42**. The ratio is three, and looked at one by one the distribution is even sharper: the listening list eliminates only one candidate in **seven** of the eight cases and seven in a single case. The connection attempt never drops below four in any case. The source of the difference is not the amount of output but **the number of answers**. The listening list can produce dozens of lines; the distinction it carries is a single binary question — is the service listening or not. The connection attempt produces a single line and carries four distinct answers: unreachable, timeout, refused, established. Each answer corresponds to a different candidate set. **A test's elimination-power ceiling is set not by the length of its output but by the number of states it distinguishes.** Which candidates the answers correspond to is used directly in diagnosis. The **unreachable** answer points to path problems and leaves four candidates together; this is the test's weakest answer, and the previous three lessons' tests do the separating. The **refused** answer says the packet reached the target and there is no listener at the target; this means everything along the path is working, and it leaves a single candidate. The **timeout** answer says the packet went out and no reply came back at all; this is the signature of a filter silently dropping it. The **established** answer says the connection opened, and surprisingly leaves two candidates standing: a silent resolver and a large packet dropped along the way. A connection being established does not mean the connection is **usable**. This last line is this lesson's counterpart of the course's second claim. The distinction between refused and timeout is a single word and eliminates seven candidates each; yet the most detailed operation that could be done to obtain the same information — recording all the traffic — produces thousands of lines. **More output does not mean better diagnosis**; what matters is how many distinct states the output corresponds to. This measure also has a reverse reading. The connection attempt's four answers are valuable because they correspond to four distinct **fault classes**; if the sheer number of answers were enough on its own, a tool that split every answer into its own error code would be unboundedly powerful. What is valuable is that the answers split the candidate set **evenly**: the four answers split eight candidates 4-1-1-2. Three of the answers come down to one or two candidates, one gets stuck at four. The answer that weakens the test is the most frequently encountered answer; `unreachable` here is exactly that, and the previous three lessons' tests exist precisely to separate the four candidates that answer leaves. #### The Inventory's Place The listening list being weak does not make it unnecessary; it changes its place. The list is a confirmation tool when the connection attempt says **refused**: the refusal says there is no listener at the target, the inventory confirms this locally and shows which bind address it is listening on. The list is also the only option in cases where the connection attempt **cannot be made** — if there is no outside access, if there is no second endpoint to run the test from, the local inventory is the only reading left. - **AY27** — The connection attempt is made from a separate endpoint that can reach the target; that endpoint's own configuration is assumed sound. The attempt having two endpoints is what raises this test's cost. - **AY28** — The distinction between timeout and refusal is assumed visible on the client side; cases where an intermediate layer changes the answer are outside the model. The inventory's second function is not diagnosis but **surface counting**. Every port listening from outside is a door opened outward; a port listening only from loopback is not. This distinction is the starting point of the firewall lessons, and it is handled there by counting exposed surface. This lesson says only this: the inventory's line count is not a health measure, and counting lines without reading the bind-address column is misleading. Its third function is comparison. Taking the inventory before and after a configuration change is the cheapest record showing what the change actually did. When a unit is restarted, the port it listens on may have changed, the bind address may have narrowed or widened; the difference between two lists gives this in a single glance. This is a measurement taken not during diagnosis but during the change, and having it in hand after a fault is born is far more valuable than searching for it after the fact. #### Candidate Set Sweep - **AY29** — The added candidate, `service-crashed`, is the case of the unit crashing, and in the model it breaks the same field as `service-not-listening`. - **AY30** — The removed candidate, `firewall-dropping`, is the candidate that alone fills the connection attempt's fourth answer. The fourth table compares the two tests under the same shake. When the candidate is added, the listening list's `not-listening` answer still eliminates seven candidates but leaves **two** behind; the same is true of the connection attempt's `refused` answer. Both tests weaken by the same amount, because both see the same field. This is an expected result and confirms the model's internal consistency. When the candidate is removed, the tests behave differently. The listening list loses only one candidate: the candidates the `not-listening` answer eliminates drop from seven to **six**, and its structure does not change. The connection attempt's answer count, on the other hand, drops from four to **three**; because the `timeout` answer now has no counterpart in the candidate list, the number of states the test distinguishes falls. Even though the worst-case remaining stays fixed at four, the distinction the test carries has shrunk. **A test's power is the number of its answers that have a counterpart in the candidate list**, and this number changes as the list changes. This is the common result of the sweep done four times across the topic. In the interface test, the answer thought to be decisive lost its decisiveness; the same happened in the route reading; in name resolution, the test became entirely worthless; here, the connection attempt lost one of its answers. In all four cases, the test's definition was never touched. The number measured is not a property of the test, it is **a joint product of the test and the candidate list**, and this is why a diagnostic method's portability depends on the portability of the candidate list. #### Summary - The local inventory's most important column is not the port number but the bind address; a port listening only from loopback is not listening at all from outside, and it appears the same way in the list. - The listening list gives **two answers** and eliminates only **one** candidate on the expected answer; a connection attempt made from outside gives **four answers** and eliminates **seven** on the same fault. - Summed across all eight faults, the listening list eliminates **14** candidates, the connection attempt **42**. The source of the difference is not output length but the number of states the test distinguishes. - The connection attempt's four answers correspond to four distinct candidate sets: unreachable leaves four candidates together, refused and timeout each leave one, and established leaves two — a connection being established does not mean it is usable. - The sweep separates the two tests: when a candidate is added, both weaken by the same amount; when a candidate is removed, the listening list's structure is preserved while the connection attempt's answer count drops from four to three. #### Next Step This topic measured four tests one by one and counted how many candidates each eliminates. The resulting table shows a single test is not enough: even the strongest leaves four candidates together in some cases, and the weakest separates a candidate no other test can. So the real question is not the value of a single test, but **the order tests should be done in**. The next topic asks this question: how many steps does the layered diagnostic method spend to resolve all eight faults, how many steps does the reverse order spend, and how much does an order that picks the most eliminating test at every step shorten this. What will be seen there is that the shortest order is **not** an applicable procedure: picking the most eliminating test requires knowing in advance which fault is real, and that is exactly the one thing the person doing the diagnosis does not know. ## Troubleshooting The step cost of the layered diagnosis method, correctly reading reachability tests, packet capture's elimination power, and the trace left by intermittent faults. ### The Layered Diagnosis Method Source: https://academia.sh/en/courses/linux-network-administration/layered-diagnosis-method Measuring diagnosis order over eight candidate causes and eight tests: the layered order spends 33 steps, the reverse order 38, and a greedy order that consults the oracle at every step spends 10; the greedy order is not a usable procedure because it requires knowing the answer, so it is an unreachable lower bound. The previous lesson worked out the local service inventory and set two tests' elimination power side by side: the listening socket list eliminated only **one** candidate cause, a connection attempt made from outside eliminated **seven**. There are now eight separate tests in hand, and all of them are runnable. This lesson's question is not which test to run, but **in what order**. The question is not an empty one. Trying whatever comes to mind first when facing a fault, reaching for the most familiar tool, or assuming last time's cause are all orders, and each carries a cost. The unit of that cost is settled too: **how many tests get run**. This lesson runs three orders over the same eight faults and counts the difference between them. One of the numbers will come out surprisingly small; why that number cannot be turned into a method is this lesson's real subject. #### The Candidate Cause Set Diagnosis is the work of narrowing a set. At the start there is one symptom — "cannot connect to the server" — and more than one cause that could produce it. The course's mock setup fixes this set as **eight candidate causes**: the interface being down, the address not being assigned, the default gateway being wrong, the route being missing, the resolver not responding, the service not listening, the firewall dropping the packet, and a large packet getting lost along the way. All eight tests operate on this same set. Each test corresponds to a tool, returns a response, and that response splits the candidate set. The **oracle** is the real fault: because we chose the mock setup, we know which fault is active. The operator does not know; all the operator has are responses. The mock setup's assumptions: there is one client, one server, and a single target service, and the candidate-cause set is limited to these eight (**SG1**); each fault breaks a single field of the healthy state, and two faults never occur at once (**SG2**); a test returns one stable response, and the process stops once the candidate count drops to one (**SG3**); the greedy order picks, at every step, the test that eliminates the most by knowing the real fault (**SG4**); every test's cost is equal and counts as one step (**SG5**). Where the list itself comes from is a separate question, and it is diagnosis's least discussed step. The candidate-cause set is derived from the symptom: which components could produce this symptom, which could not. Keeping the set narrow lowers the number of tests but grows the risk of leaving out the real cause; keeping it wide does the opposite. The eight-candidate mock setup is a **closed-world** assumption: the real fault is taken to be on the list. When the closed world breaks, what breaks is not the measurement itself but its interpretation — every test finishes, one candidate remains, and that candidate is wrong. A diagnosis order finishing does not mean it finished correctly. Another source that narrows the set is the logs measured in the system administration course. If a service was restarted, a unit failed, or a configuration change was logged, this puts some candidates forward before any test is run. The role of logs here is not evidence but a **prior**: it affects the ordering, it does not eliminate a candidate. Elimination only happens through a test's response. Protocol layers themselves, handshakes, and routing protocols are the subject of the Computer Networks curriculum and are not built here. This course looks at a single machine's configuration and diagnosis. Namespaces and container networking are not covered either; they are left to the Kernel Interfaces and Isolation course. #### The Layered Order and Its Two Rivals **Layered diagnosis** orders tests by dependency: first the interface, then the address and route, then local-network reach, then remote reach, then name resolution, and lastly the service itself. The reasoning is intuitive — if the bottom is broken, the result at the top is already meaningless. The **reverse order** reads the same list backward: it first tries connecting to the service, then works its way down. This too is a defensible order; an operator who thinks most faults sit on the service side starts right there. The **greedy order** picks, at every step, the test that leaves the fewest candidates behind. All three orders use the same tests; the dump below shows which commands these correspond to and it **has not been run**. The measurement happens in the block that follows, and it runs all three orders over the same eight faults. ```text # example dump , has not been run $ ip -brief link show # is the interface up $ ip -brief addr show # is an address assigned $ ip route show # default gateway and route $ ping -c 3 # echo to the local network $ getent hosts # name resolution $ ss -ltn # listening ports $ nc -z -w 3 # connection attempt ``` ```python HEALTHY = {"interface_up": True, "address_set": True, "gateway_correct": True, "route_present": True, "resolver_responds": True, "service_listening": True, "firewall_passes": True, "path_ok": True} FAULT = { # SG2: each fault breaks a SINGLE field "interface-down": {"interface_up": False, "address_set": False}, "no-address": {"address_set": False}, "wrong-gateway": {"gateway_correct": False}, "missing-route": {"route_present": False}, "resolver-silent": {"resolver_responds": False}, "service-not-listening": {"service_listening": False}, "firewall-dropping": {"firewall_passes": False}, "large-packet-lost": {"path_ok": False}, } CANDIDATES = tuple(FAULT) def state(fault=None): d = dict(HEALTHY) if fault: d.update(FAULT[fault]) return d def path(d): # interface + address + gateway + route together return (d["interface_up"] and d["address_set"] and d["gateway_correct"] and d["route_present"]) TEST = { "interface": lambda d: ("open-addressed" if d["interface_up"] and d["address_set"] else "open-unaddressed" if d["interface_up"] else "down"), "route": lambda d: ("table-empty" if not d["address_set"] else "complete" if d["route_present"] else "no-default"), "gateway-echo": lambda d: ("response" if d["interface_up"] and d["address_set"] else "no-response"), "remote-echo": lambda d: "response" if path(d) else "no-response", "name-lookup": lambda d: "resolved" if d["resolver_responds"] else "unresolved", "listening": lambda d: "listening" if d["service_listening"] else "not-listening", "connection": lambda d: ("unreachable" if not path(d) else "timeout" if not d["firewall_passes"] else "refused" if not d["service_listening"] else "established"), "large-packet": lambda d: ("passes" if path(d) and d["path_ok"] else "blocked"), } def eliminate(candidates, s, response): return tuple(a for a in candidates if TEST[s](state(a)) == response) def eliminates(candidates, s, real): remaining = eliminate(candidates, s, TEST[s](state(real))) return len(candidates) - len(remaining), len(remaining) def cost(order, real, candidates=CANDIDATES): # SG3: stop once one candidate remains remaining, step = tuple(candidates), 0 for s in order: if len(remaining) <= 1: break step += 1 remaining = eliminate(remaining, s, TEST[s](state(real))) return step def greedy(real, candidates=CANDIDATES): # SG4: looks at the oracle every step remaining, chosen = tuple(candidates), [] while len(remaining) > 1 and len(chosen) < len(TEST): best, best_s = None, None for s in TEST: if s not in chosen: k = len(eliminate(remaining, s, TEST[s](state(real)))) if best is None or k < best: best, best_s = k, s chosen.append(best_s) remaining = eliminate(remaining, best_s, TEST[best_s](state(real))) return chosen LAYERED = ["interface", "route", "gateway-echo", "remote-echo", "name-lookup", "listening", "connection", "large-packet"] REVERSE = list(reversed(LAYERED)) ORACLE = "firewall-dropping" print("candidate faults:", len(CANDIDATES), "| tests:", len(TEST), "| oracle:", ORACLE) print("test response eliminated left") for s in LAYERED: e, k = eliminates(CANDIDATES, s, ORACLE) print(f" {s:14s}{TEST[s](state(ORACLE)):16s}{e:6d} {k:6d}") print() print("fault layered reverse greedy greedy order") for a in CANDIDATES: g = greedy(a) print(f" {a:26s}{cost(LAYERED, a):6d}{cost(REVERSE, a):6d}" f"{len(g):8d} {', '.join(g)}") print("total steps: layered", sum(cost(LAYERED, a) for a in CANDIDATES), "| reverse", sum(cost(REVERSE, a) for a in CANDIDATES), "| greedy", sum(len(greedy(a)) for a in CANDIDATES)) print() print("candidate set sweep -- eliminated by connection test as the set grows") group = [ORACLE] for a in CANDIDATES: if a not in group: group.append(a) e, k = eliminates(tuple(group), "connection", ORACLE) print(f" candidate {len(group)} added {a:26s} eliminated {e} left {k}") print("layered total when one candidate is removed (33 at 8 candidates)") for out in CANDIDATES: rest = tuple(x for x in CANDIDATES if x != out) print(f" removed {out:26s} {sum(cost(LAYERED, a, rest) for a in rest):3d}") ``` ``` candidate faults: 8 | tests: 8 | oracle: firewall-dropping test response eliminated left interface open-addressed 2 6 route complete 3 5 gateway-echo response 2 6 remote-echo response 4 4 name-lookup resolved 1 7 listening listening 1 7 connection timeout 7 1 large-packet passes 5 3 fault layered reverse greedy greedy order interface-down 1 8 1 interface no-address 1 8 1 interface wrong-gateway 4 7 2 remote-echo, route missing-route 2 7 1 route resolver-silent 5 2 1 name-lookup service-not-listening 6 2 1 listening firewall-dropping 7 2 1 connection large-packet-lost 7 2 2 connection, name-lookup total steps: layered 33 | reverse 38 | greedy 10 candidate set sweep -- eliminated by connection test as the set grows candidate 2 added interface-down eliminated 1 left 1 candidate 3 added no-address eliminated 2 left 1 candidate 4 added wrong-gateway eliminated 3 left 1 candidate 5 added missing-route eliminated 4 left 1 candidate 6 added resolver-silent eliminated 5 left 1 candidate 7 added service-not-listening eliminated 6 left 1 candidate 8 added large-packet-lost eliminated 7 left 1 layered total when one candidate is removed (33 at 8 candidates) removed interface-down 32 removed no-address 32 removed wrong-gateway 29 removed missing-route 31 removed resolver-silent 28 removed service-not-listening 27 removed firewall-dropping 25 removed large-packet-lost 25 ``` #### Oracle, Test, Eliminated Candidate Three numbers sit side by side. **Oracle:** the real fault is `firewall-dropping`, that is, the firewall is silently dropping the packet. **Test:** the connection attempt returns `timeout`. **Eliminated candidate:** this single response eliminates **7** of the eight candidates and leaves **1** behind — the fault is resolved with a single test. Under the same oracle, the name-lookup test returns `resolved`, the listening test returns `listening`; each eliminates only **1** candidate and leaves **7** behind. The resulting ratio is the lesson's first reading: of two tests done with equal effort, one does **seven times** the work of the other. A test's value is not in the response it gives, it is in how many candidates that response drops. A test whose response is correct but eliminates no candidate counts as **unmeasured**. The total-step table is the second reading. When each of the eight faults is made the real fault in turn, the layered order spends a total of **33** steps, the reverse order **38**, the greedy order **10**. The layered order runs **3.3 times** as many tests as the optimum. The reverse order's loss is in the distribution: it finishes the four service-side faults in 2 steps, but spends the full 8 steps on the two interface-side faults. #### Why the Greedy Order Is Not a Procedure The number ten can pull the reader toward a wrong conclusion: the rule "pick the test that eliminates the most" looks like a usable method. It is not. For the rule to work, at every step, it has to compute what response every test will give under the real fault, which is **not yet known**. The line in the block that does this is the `TEST[s](state(real))` call, and `real` is exactly what the diagnosis is trying to find. In other words, the greedy order **takes the oracle as input**. Someone who knows the answer can pick the shortest path; someone who does not know it cannot make that choice. So the number 10 is not a target, it is an **unreachable lower bound**. It has one job in the measurement — it gives a scale showing how much extra work the layered order does — but it is never presented anywhere as a usable order. This distinction is often confused in diagnosis writing. The sentence "an experienced operator looks straight at the right place" is the human form of the greedy order, and it carries the same flaw: experience is knowing the distribution of past faults, not the current one. When the past distribution matches today's fault, the shortcut works; when it does not, the operator lives through the reverse order's worst case — running all eight tests while the interface is down, for instance. #### The Layered Order's Defense The layered method's defense is not speed; 33 steps cost three times as much as 10, and that gap is real. The defense is this: **the layered order works without an oracle.** The order is fixed before the fault occurs and runs in the same order whichever fault is active. Its only input is the tests' responses. The second defense is that the intermediate information the order gives is **cumulative**. When the interface test returns `open-addressed`, it does not just eliminate two candidates; it also fixes how subsequent tests get interpreted. The reverse order has no such accumulation: once it is seen that the service cannot be connected to, the remaining four candidates are still unsplit, and every test going downward is interpreted from scratch. The third defense is protection against a wrong diagnosis. While the reverse order runs, the urge arises to fix something on the service side — restarting the service, loosening a rule — and such fixes made while the layer beneath is broken change the system's state. The layered order proves the bottom is sound before moving on to a fix. A fourth point is a practical warning: the most expensive faults sit at the bottom of the table. `firewall-dropping` and `large-packet-lost` each cost **7** steps in the layered order. For an operator looking for a shortcut, the right move is not to break the order but to **move the connection attempt earlier in it**: because it returns four separate responses, it is the strongest single splitter on its own and can be moved up without breaking the order's logic. #### Candidate Set Sweep There is no randomness in the mock setup; every number is a full count. How much the results depend on the mock setup is therefore tested not with a second seed but by **changing the candidate set**. As the set grows, the number of candidates the connection test eliminates climbs from 1 to 7, and what is left is always 1. What follows from this matters: **the elimination count is not a property of the tool alone**, it is also a measure of the candidate set. The sentence "this tool eliminates seven candidates" is meaningful only once an eight-candidate set has been defined. Comparing tools without writing down the candidate list is writing a fraction with no denominator. When one candidate is removed, the layered total drops from 33 to somewhere between 25 and 32. Where the drop lands depends on where the removed candidate sits in the order: removing `interface-down`, which sits at the front, drops the total by only 1 step (32); removing `firewall-dropping` or `large-packet-lost`, at the back, drops it by 8 steps (25). What determines the layered order's cost is not the candidate count but **where in the order** the candidates sit. The reverse also holds: a newly added deep candidate can grow the total by as much as eight steps. #### Summary - Diagnosis is the work of narrowing a set; a test's value is measured not by its response but by the **number of candidate causes it eliminates**, and a test whose elimination count is not written down counts as unmeasured. - In the firewall-drop scenario, the connection attempt eliminates **7** candidates and resolves it alone; name lookup and the listening list each eliminate only **1**. - Total step cost across all eight faults: layered order **33**, reverse order **38**, oracle-assisted greedy order **10**. - The greedy order **is not a procedure**: picking the most eliminating test at every step requires knowing the real fault, so 10 is an unreachable lower bound. - The layered method's defense is not speed but that it **works without an oracle**; its order is fixed before the fault and takes only test responses as input. - Elimination count depends on the candidate set: as the set grows from 2 to 8, what the connection test eliminates climbs from 1 to 7. #### Next Step The layered order's first four steps are reachability tests, and the table's weakest-looking entries sit right there: echo to the gateway eliminates only 2 candidates, echo to the remote address 4. These tests' real problem is not their weakness, it is that they are **misread**: no response fits not one cause but several, and a response arriving does not say reach is sound either. The next lesson opens up the silence returned by echo and path-test tools and counts how many separate candidate causes a "no response" state corresponds to. ### Reachability Tests Source: https://academia.sh/en/courses/linux-network-administration/reachability-tests Reading echo and path-test tools correctly: no response from a remote address fits four separate candidate causes, three reachability tests plus a connection attempt together still cannot split two candidate pairs, and what splits them is reading the configuration. The previous lesson measured the diagnosis order and showed that two tests in the layered order's first steps looked weak: echo to the gateway eliminated only 2 candidates, echo to the remote address 4. These tests are diagnosis's most commonly used tools, and their weakness is not the real problem. The real problem is that they are **misread**. The misreading runs in two directions. In the first direction, silence is assumed to say too much: when no response arrives, the conclusion "the machine is down" or "there is no network" is reached, even though more than one cause produces the same silence. In the second direction, a response is assumed to say too much: when a response arrives, "reach is fine" is said, even though the layer that responded is not the layer being reached. This lesson meets both misreadings with numbers. #### The Echo Test's Two-Valued Response An echo test sends a small request to an address and waits for a response in return. The mock setup reduces this to a two-valued response: `response` or `no-response` (**SG6**). Latency, loss rate, and response order are not measured in this lesson; what is measured is how many candidate causes the silence **fits**. There are two separate echo tests, and they look at different distances. **Gateway echo** asks whether we can reach the local network at all; for a response, the interface being up and the address being assigned is enough. **Remote echo** asks about the path all the way to the target; for a response, the gateway also has to be correct and the route has to exist. The second test requires everything the first one requires, plus two more conditions. This nesting directly explains why the silence fits more than one cause. Name resolution is kept out of these tests: echo is done with the address, not the name. An echo done by name mixes two mechanisms into a single response and blurs the cause of the silence by one more candidate. The protocol itself is the subject of the Computer Networks curriculum and is not built here. #### Path Test: The Same Test Repeated at Increasing Distance **Traceroute-style path testing** is the common name for tools that show the path to a target along with its intermediate stops. What it does is repeat the same reachability question at increasing distances: first one hop out, then two hops out. In the mock setup this corresponds to running gateway echo and remote echo back to back (**SG7**); the intermediate stops' individual behavior is not modeled separately. A third reachability test is a **large-packet** attempt that is not allowed to fragment (**SG8**). A path where small packets get through but large ones get lost looks sound to an echo test. This split comes from echo sending a small request: the packet size a test uses determines the fault it cannot see. A fourth boundary needs to be written up front too: an echo response being deliberately withheld is not in the candidate-cause set (**SG9**). A target can be configured not to respond to echo at all, and in that case the silence corresponds to no fault on the list. This is the price of the closed-world assumption: a cause outside the list gives the same response as a cause inside it. The most commonly misread part of path-test output is **an intermediate stop's silence**. No response from a stop does not show that stop is broken; intermediate stops are not obligated to respond to requests not addressed to them, and most treat it as low-priority work. The only meaningful line in a path test is the **last** one: at what distance the path goes completely silent. Empty lines in the middle eliminate no candidate, they only add to the line count. The same mistake runs in reverse too: every stop responding while the target stays silent suggests the path is sound and the fault sits at the target, but the return direction is a separate path, and the outbound direction being sound does not prove the return is. The mock setup's reduction to a two-valued response is not a convenience, it is a deliberate boundary. A real echo test also says how many of three requests came back, and "3 lost out of 3" is not the same thing, diagnostically, as "1 lost out of 3": the first points to a persistent fault, the second to an intermittent one. The mock setup does not cover this second case; intermittent faults are the fourth lesson's subject. The point to hold onto here is this: every reading that drops the loss rate and reduces the response to two values shows an intermittent fault as either fully sound or fully broken. ```text # example dump , has not been run $ ping -c 3 # one hop out $ ping -c 3 # all the way to the target $ tracepath # repetition at increasing distance $ ping -c 3 -M do -s 1400 # attempt that disallows fragmentation ``` ```python HEALTHY = {"interface_up": True, "address_set": True, "gateway_correct": True, "route_present": True, "resolver_responds": True, "service_listening": True, "firewall_passes": True, "path_ok": True} FAULT = { "interface-down": {"interface_up": False, "address_set": False}, "no-address": {"address_set": False}, "wrong-gateway": {"gateway_correct": False}, "missing-route": {"route_present": False}, "resolver-silent": {"resolver_responds": False}, "service-not-listening": {"service_listening": False}, "firewall-dropping": {"firewall_passes": False}, "large-packet-lost": {"path_ok": False}, } CANDIDATES = tuple(FAULT) def state(a=None): d = dict(HEALTHY) if a: d.update(FAULT[a]) return d def path(d): return (d["interface_up"] and d["address_set"] and d["gateway_correct"] and d["route_present"]) TEST = { # five of the common definition's eight used in this lesson "gateway-echo": lambda d: ("response" if d["interface_up"] and d["address_set"] else "no-response"), "remote-echo": lambda d: "response" if path(d) else "no-response", "large-packet": lambda d: ("passes" if path(d) and d["path_ok"] else "blocked"), "connection": lambda d: ("unreachable" if not path(d) else "timeout" if not d["firewall_passes"] else "refused" if not d["service_listening"] else "established"), "route": lambda d: ("table-empty" if not d["address_set"] else "complete" if d["route_present"] else "no-default"), } ECHO = ["gateway-echo", "remote-echo", "large-packet"] # SG7: traceroute-style def groups(kit, candidates=CANDIDATES): o = {} for a in candidates: o.setdefault(tuple(TEST[s](state(a)) for s in kit), []).append(a) return o print("how many candidate causes a response fits") for s in ECHO + ["connection"]: for response, fits in sorted(groups([s]).items()): print(f" {s:14s} {response[0]:12s} {len(fits)} {', '.join(sorted(fits))}") print() ORACLE = "wrong-gateway" print("oracle:", ORACLE, "| tests applied in order") remaining = CANDIDATES for s in ECHO + ["connection", "route"]: response = TEST[s](state(ORACLE)) new = tuple(a for a in remaining if TEST[s](state(a)) == response) print(f" {s:14s} {response:12s} eliminated {len(remaining) - len(new)} left {len(new)}" f" {', '.join(sorted(new))}") remaining = new print() print("test kit groups unsplit largest group") for kit in (["gateway-echo"], ["remote-echo"], ["gateway-echo", "remote-echo"], ECHO, ["connection"], ECHO + ["connection"]): o = groups(kit) split = [v for v in o.values() if len(v) > 1] print(f" {'+'.join(kit):45s} {len(o):4d} {len(split):12d}" f" {max(len(v) for v in o.values()):14d}") print() print("candidate set sweep: how many candidates 'remote-echo = no-response' fits") kit = ["interface-down"] for a in CANDIDATES: if a not in kit: kit.append(a) fits = [x for x in kit if TEST["remote-echo"](state(x)) == "no-response"] print(f" candidate {len(kit)} added {a:26s} fits {len(fits)}") ``` ``` how many candidate causes a response fits gateway-echo no-response 2 interface-down, no-address gateway-echo response 6 firewall-dropping, large-packet-lost, missing-route, resolver-silent, service-not-listening, wrong-gateway remote-echo no-response 4 interface-down, missing-route, no-address, wrong-gateway remote-echo response 4 firewall-dropping, large-packet-lost, resolver-silent, service-not-listening large-packet blocked 5 interface-down, large-packet-lost, missing-route, no-address, wrong-gateway large-packet passes 3 firewall-dropping, resolver-silent, service-not-listening connection established 2 large-packet-lost, resolver-silent connection refused 1 service-not-listening connection timeout 1 firewall-dropping connection unreachable 4 interface-down, missing-route, no-address, wrong-gateway oracle: wrong-gateway | tests applied in order gateway-echo response eliminated 2 left 6 firewall-dropping, large-packet-lost, missing-route, resolver-silent, service-not-listening, wrong-gateway remote-echo no-response eliminated 4 left 2 missing-route, wrong-gateway large-packet blocked eliminated 0 left 2 missing-route, wrong-gateway connection unreachable eliminated 0 left 2 missing-route, wrong-gateway route complete eliminated 1 left 1 wrong-gateway test kit groups unsplit largest group gateway-echo 2 2 6 remote-echo 2 2 4 gateway-echo+remote-echo 3 3 4 gateway-echo+remote-echo+large-packet 4 3 3 connection 4 2 4 gateway-echo+remote-echo+large-packet+connection 6 2 2 candidate set sweep: how many candidates 'remote-echo = no-response' fits candidate 2 added no-address fits 2 candidate 3 added wrong-gateway fits 3 candidate 4 added missing-route fits 4 candidate 5 added resolver-silent fits 4 candidate 6 added service-not-listening fits 4 candidate 7 added firewall-dropping fits 4 candidate 8 added large-packet-lost fits 4 ``` #### Oracle, Test, Eliminated Candidate Three numbers sit side by side. **Oracle:** the real fault is `wrong-gateway`, that is, the default gateway is set wrong. **Test:** remote echo returns `no-response`. **Eliminated candidate:** this response eliminates **4** of eight candidates and leaves **4** behind. The silence fitting four candidates is this lesson's first number. The sentence "no response from the remote address" is on its own consistent with **all** of `interface-down`, `no-address`, `wrong-gateway`, and `missing-route`. Reading the silence as "network's down" lumps these four together as if they were one cause; fixing each of the four is different, and picking the wrong fix does not work. When gateway echo stays silent, the picture is narrower: only **2** candidates fit (`interface-down`, `no-address`). A practical reading follows from this. **Nearby silence is more informative than distant silence**, because it depends on fewer conditions. This is exactly the path test's real function too: finding at what distance silence begins, to narrow the number of candidates that fit. #### What a Response Does Not Say The second misreading is sneakier. Echo returning a response eliminates four candidates but leaves **4** behind: `resolver-silent`, `service-not-listening`, `firewall-dropping`, and `large-packet-lost`. That is, the sentence "the machine is responding" does not mean the service is reachable, and four separate faults are entirely consistent with that sentence. What makes the difference is that silence and **being refused** are not the same thing. Two of the connection test's four responses each pin down a single candidate: `refused` fits only `service-not-listening`, `timeout` only `firewall-dropping`. Being refused is a response, and the responding side also proves the path is sound; silence is ambiguous about the state of both the path and the target. So what diagnosis looks for is a test that can **turn silence into a response**. The large-packet attempt splits off one of these four: in the `large-packet-lost` scenario, a small echo gets through, a large packet does not. This is diagnosis's most commonly overlooked signature, because it is the one place two tests contradict each other — one says sound, the other broken, and both are correct. The step-by-step table shows how this order runs. With the oracle at `wrong-gateway`, gateway echo eliminates 2 candidates, remote echo eliminates 4; after that, the large-packet attempt eliminates **0**, and so does the connection attempt. A test being powerful is not enough: **re-eliminating an already-eliminated candidate is not elimination.** The connection attempt eliminated seven candidates in the previous lesson; here it eliminates zero, because the tests that ran before it had already made the same split. #### The Wall Reachability Tests Hit The last table is the lesson's harshest result. Gateway echo alone splits the eight candidates into 2 groups, and the largest group holds 6. Remote echo alone gives 2 groups, largest 4. Together they produce 3 groups, and **all three groups hold more than one candidate**: these two tests cannot pin down even a single candidate on their own. Adding the large-packet attempt brings the group count to 4 and the largest group down to 3; the number of unsplit groups is still **3**. Adding the connection attempt too produces 6 groups, and **2** unsplit pairs remain: `interface-down` with `no-address`, and `wrong-gateway` with `missing-route`. These two pairs stay unsplit no matter how many reachability tests are run. The reason is structural: both members of each pair cut **the same path at the same place**, so every test that looks at the path gives the same response. What splits them is not looking at the path, it is **reading the configuration**. The table's last row does this: the routing-table test returns `complete`, eliminates the last candidate between `wrong-gateway` and `missing-route`, and the diagnosis drops to one candidate. This is where the course's second claim shows its first face: **adding more tests does not add elimination power.** Three reachability tests do not do what a single configuration reading does. What is needed is not multiplying tests but **changing their kind**. The reason these tools get used this much is hidden in the same table: they are cheap. An echo test takes seconds, is a one-line command, changes nothing on the target, and its failure is harmless. Cheapness makes a tool the first choice; **elimination power** is what makes it the tool that finishes the diagnosis, and the two are not the same measure. The measurement counted every test as one step (**SG5**), that is, decisions were counted, not seconds. Measured in seconds, echo tests would look even more favorable, and it is exactly this appearance that leads to dozens of repeated echo tests once diagnosis stalls. Repeating the same test eliminates no new candidate; the information gained after the first run is fixed. #### Candidate Set Sweep The sweep shows how much the number of candidates the silence fits depends on the set. As the set grows from `interface-down` up to eight, the number of candidates fitting `remote-echo = no-response` climbs 1, 2, 3, 4, and **stops at four**: the four candidates added after that sit in an upper layer and produce a response to remote echo. The table can also be read backward. Removing a candidate from the list is the same as reading the rows upward: with `missing-route` removed, the candidates fitting the silence drop from 4 to 3; with `wrong-gateway` also removed, to 2. Removing always looks like a gain, because the silence narrows; but if the removed candidate is the real fault, the diagnosis ends with zero candidates, and this shows up nowhere in the table. The result cuts both ways. If the candidate list is short, silence looks more informative — in a three-candidate list, "no response" fits two candidates — but that information is bought at the cost of causes left outside the list. As the candidate list grows, silence's discriminating power **neither grows nor shrinks** past a certain point: as long as added candidates give a different response, silence's meaning stays fixed. What determines a test's power is not the candidate count, it is **which response the candidates fall into**. #### Summary - No response from a remote address fits **4** separate candidate causes; no response from the gateway fits **2**. Nearby silence is more informative because it depends on fewer conditions. - Echo returning a response also leaves **4** candidates standing; "the machine is responding" does not say the service is reachable. - A path test is the same reachability question repeated at increasing distance; the large-packet attempt splits off the one fault a small echo cannot see. - Three reachability tests together cannot pin down even a single candidate; adding the connection attempt leaves **2** unsplit pairs, and only reading the configuration splits them. - Re-eliminating an already-eliminated candidate is not elimination: the connection attempt eliminates **0** candidates in this order; repeating the same test eliminates no new one either. - Being refused is a response and pins down a single candidate; silence is ambiguous about both the path and the target. #### Next Step When reachability tests hit a wall, the next step that comes to mind is usually the same one: looking at the traffic itself. Packet capture does not show a summary of responses, it shows every frame crossing the wire, and for that reason it is counted the "most detailed" tool. The next lesson measures that expectation: it sets the number of lines a capture produces against the number of candidate causes it eliminates, and compares elimination power per line against the connection attempt's. That packet capture is a diagnosis tool, not a procedure for seizing someone else's traffic, is also written there, along with its boundary. ### Packet Capture Source: https://academia.sh/en/courses/linux-network-administration/packet-capture Measuring capture's elimination power: an unfiltered capture produces 14400 lines and eliminates 6 candidates without finishing the diagnosis; a one-line connection attempt eliminates 7 and finishes it. Elimination per line comes out to 0.0004 against 7.0000. The previous lesson showed reachability tests hitting a wall: three echo tests and a connection attempt together still could not split two candidate pairs. At a stall like this, the next step that comes to mind is usually the same one — not looking at summaries, but at **the traffic crossing the wire itself.** Packet capture does this, and for that reason it is counted the toolbox's most detailed tool. This lesson measures that expectation. The number of lines capture produces is set against the number of candidate causes it eliminates, and the ratio between the two is computed. The result is the course's second claim in its harshest form: **more output does not mean a better diagnosis.** The lesson's second half writes up a boundary that follows from the same measurement: capture is a **diagnosis** tool, and no use of it beyond diagnosis is covered in this course. #### What Capture Sees Versus What It Says Capture records the frames crossing an interface. What it sees is genuinely a lot: every frame's time, size, direction, and header fields. What it says is only as much as diagnosis can use, and the operator reduces these thousands of lines to **a single verdict**: which directions the traffic is flowing in. The common definition's eight tests are untouched; capture is added in this lesson as a **ninth** test (**SG10**). Capture is done on the local interface and only over the flow under diagnosis; its response is reduced to three values (**SG11**): `no-packets`, `outbound-no-return`, `two-way`. Every captured frame produces one line (**SG12**). Over a 600-second window, the distribution of frames crossing the interface is a mock value and totals 14400 (**SG13**). A filter lowers the number of captured lines but does not change the response (**SG14**); the narrowest reading is the flow's first six frames (**SG15**). Capture is up and running before the fault begins (**SG16**). The three responses correspond directly to the healthy state. If the interface is down, the address is missing, or the route is missing, the local stack cannot produce frames and the capture is empty. If the gateway is wrong or the firewall is dropping, frames go out and get no answer. In the remaining cases, two-way traffic is visible. The meaning of protocol headers and handshakes is the Computer Networks curriculum's subject and is not opened here. ```text # example dump , has not been run $ tcpdump -i -n -c 200 host and port $ tcpdump -i -n -w /tmp/diag.pcap host and port $ tcpdump -r /tmp/diag.pcap -n | wc -l ``` ```python HEALTHY = {"interface_up": True, "address_set": True, "gateway_correct": True, "route_present": True, "resolver_responds": True, "service_listening": True, "firewall_passes": True, "path_ok": True} FAULT = { "interface-down": {"interface_up": False, "address_set": False}, "no-address": {"address_set": False}, "wrong-gateway": {"gateway_correct": False}, "missing-route": {"route_present": False}, "resolver-silent": {"resolver_responds": False}, "service-not-listening": {"service_listening": False}, "firewall-dropping": {"firewall_passes": False}, "large-packet-lost": {"path_ok": False}, } CANDIDATES = tuple(FAULT) def state(a=None): d = dict(HEALTHY) if a: d.update(FAULT[a]) return d def path(d): return (d["interface_up"] and d["address_set"] and d["gateway_correct"] and d["route_present"]) TEST = { "interface": lambda d: ("open-addressed" if d["interface_up"] and d["address_set"] else "open-unaddressed" if d["interface_up"] else "down"), "connection": lambda d: ("unreachable" if not path(d) else "timeout" if not d["firewall_passes"] else "refused" if not d["service_listening"] else "established"), # SG10: the common definition's eight tests are untouched , capture is the NINTH test "capture": lambda d: ( "no-packets" if not (d["interface_up"] and d["address_set"] and d["route_present"]) else "outbound-no-return" if not (d["gateway_correct"] and d["firewall_passes"]) else "two-way"), } TRAFFIC = { # SG13: mock frame counts over a 600-second window "unrelated-neighbor": 9000, "name-query": 1800, "management": 2400, "target-other-port": 1080, "target-flow": 120, } FILTER = { # SG14: a filter lowers lines , NOT the response "no filter": list(TRAFFIC), "target address": ["target-other-port", "target-flow"], "target address + port": ["target-flow"], } def lines(name): return sum(TRAFFIC[k] for k in FILTER[name]) def groups(kit, candidates=CANDIDATES): o = {} for a in candidates: o.setdefault(tuple(TEST[s](state(a)) for s in kit), []).append(a) return o def eliminates(kit, real, candidates=CANDIDATES): pattern = tuple(TEST[s](state(real)) for s in kit) remaining = groups(kit, candidates)[pattern] return len(candidates) - len(remaining), len(remaining) ORACLE = "firewall-dropping" print("fault capture connection") for a in CANDIDATES: print(f" {a:26s}{TEST['capture'](state(a)):21s}{TEST['connection'](state(a))}") print() print("oracle:", ORACLE) for s in ("capture", "connection", "interface"): e, k = eliminates([s], ORACLE) print(f" {s:9s} response {TEST[s](state(ORACLE)):20s} eliminated {e} left {k}") print() print("line count and eliminated candidates per line (oracle fixed)") for name in FILTER: n = lines(name) e, _ = eliminates(["capture"], ORACLE) print(f" capture / {name:26s} lines {n:6d} eliminated {e} per line {e / n:.4f}") FIRST = 6 # SG15: the flow's first 6 frames e, _ = eliminates(["capture"], ORACLE) print(f" capture / {'first frames of flow':26s} lines {FIRST:6d} eliminated {e}" f" per line {e / FIRST:.4f}") eb, _ = eliminates(["connection"], ORACLE) print(f" connection attempt{'':20s} lines {1:6d} eliminated {eb} per line {eb / 1:.4f}") print() print("test kit groups unsplit singleton largest group") for kit in (["capture"], ["connection"], ["interface"], ["connection", "capture"], ["connection", "interface"]): o = groups(kit) print(f" {'+'.join(kit):25s} {len(o):4d} {len([v for v in o.values() if len(v) > 1]):12d}" f" {len([v for v in o.values() if len(v) == 1]):6d} {max(len(v) for v in o.values()):14d}") print() print("candidate set sweep: what capture and connection eliminate (set growing)") kit = [ORACLE] for a in CANDIDATES: if a not in kit: kit.append(a) ec, kc = eliminates(["capture"], ORACLE, tuple(kit)) eb, kb = eliminates(["connection"], ORACLE, tuple(kit)) print(f" candidate {len(kit)} capture eliminated {ec} left {kc}" f" connection eliminated {eb} left {kb}") ``` ``` fault capture connection interface-down no-packets unreachable no-address no-packets unreachable wrong-gateway outbound-no-return unreachable missing-route no-packets unreachable resolver-silent two-way established service-not-listening two-way refused firewall-dropping outbound-no-return timeout large-packet-lost two-way established oracle: firewall-dropping capture response outbound-no-return eliminated 6 left 2 connection response timeout eliminated 7 left 1 interface response open-addressed eliminated 2 left 6 line count and eliminated candidates per line (oracle fixed) capture / no filter lines 14400 eliminated 6 per line 0.0004 capture / target address lines 1200 eliminated 6 per line 0.0050 capture / target address + port lines 120 eliminated 6 per line 0.0500 capture / first frames of flow lines 6 eliminated 6 per line 1.0000 connection attempt lines 1 eliminated 7 per line 7.0000 test kit groups unsplit singleton largest group capture 3 3 0 3 connection 4 2 2 4 interface 3 1 2 6 connection+capture 5 2 3 3 connection+interface 6 2 4 2 candidate set sweep: what capture and connection eliminate (set growing) candidate 2 capture eliminated 1 left 1 connection eliminated 1 left 1 candidate 3 capture eliminated 2 left 1 connection eliminated 2 left 1 candidate 4 capture eliminated 2 left 2 connection eliminated 3 left 1 candidate 5 capture eliminated 3 left 2 connection eliminated 4 left 1 candidate 6 capture eliminated 4 left 2 connection eliminated 5 left 1 candidate 7 capture eliminated 5 left 2 connection eliminated 6 left 1 candidate 8 capture eliminated 6 left 2 connection eliminated 7 left 1 ``` #### Oracle, Test, Eliminated Candidate Three numbers sit side by side. **Oracle:** the real fault is `firewall-dropping`. **Test:** unfiltered capture returns `outbound-no-return`. **Eliminated candidate:** this response eliminates **6** of eight candidates and leaves **2**; the diagnosis does not finish. Under the same oracle, a one-line connection attempt returns `timeout`, eliminates **7** candidates, and leaves **1**; the diagnosis finishes. The gap is a single candidate, but that candidate's name is `wrong-gateway`, and capture carries it all the way to the end. The reason is simple: what capture sees is the fact "it went out, it did not come back"; it cannot see **why it did not come back.** The gateway being wrong and the firewall dropping look identical to an observer watching the interface. The same blindness sits at the other end of the table too. A `two-way` response fits three candidates at once: with the resolver silent, with the service not listening, and with a large packet lost, two-way traffic is visible on the interface in all three cases. An operator reading the recording says "traffic is flowing," and that is true; the user still cannot reach the service. What capture can say is **that frames exist**, not that the exchange succeeded. The same three candidates fall into two separate responses in the four-valued connection attempt. The group table sums this up on its own. Capture splits eight candidates into 3 groups and **no group is left with a single candidate**: capture alone cannot pin down a single fault. The connection attempt produces 4 groups, and **2** of them hold a single candidate. A one-line response does what a fourteen-thousand-line recording cannot. #### Elimination Per Line The line-count column gives the real comparison. Unfiltered capture produces **14400** lines and eliminates 6 candidates: **0.0004** candidates per line. Filtered by target address, lines drop to 1200; filtered by target address and port, to 120; the eliminated candidate count **does not change**, and elimination per line becomes 0.0050 and 0.0500. Reading only the flow's first six frames brings the ratio up to 1.0000. The connection attempt's per-line ratio is **7.0000**. Two readings follow. The first is the claim itself, directly: growing the line count from six to 14400, that is, **2400 times**, does not raise elimination power **at all**. The information drawn from capture's 14400 lines is the same as the information drawn from that same capture's first six lines. Extra lines eliminate no new candidate; they only add to reading time, file size, and the chance of latching onto the wrong pattern. Extra lines have a second cost too, and it sits outside the measurement. Capture has to process every frame it records; once the rate rises, the recording itself starts **dropping** frames. The tool producing the most output is exactly the tool that can lose evidence at the moment the load is highest, and a dropped frame does not show up as a gap in the recording, it shows up as an event that never happened. A narrow filter lowers this risk too: when fewer frames need recording, the odds of a drop fall. So a filter does not just make reading easier, it also protects the recording's **completeness**. The second reading is what a filter does. A filter does not raise elimination power, it lowers **effort** (**SG14**). This distinction is commonly confused in diagnosis: an operator looking with a narrow filter reaches a conclusion faster, but the conclusion reached is no more certain than the one reached with a wide filter. A filter is not a diagnostic decision, it is a reading convenience. A filter's real risk runs the other way: a badly set filter can leave out the frames that are the only evidence of the fault, and the recording says "no packets at all." This is a fourth cause producing the `no-packets` response, and it is not in the candidate set. #### Where Capture Actually Earns Its Keep The measurement does not show capture is unnecessary; it shows **where it belongs**. Used together with the connection attempt, group count climbs from 4 to 5, single-candidate group count from 2 to 3. Capture splits off a spot the connection attempt could not: within the four candidates under the `unreachable` response, it marks the `wrong-gateway` one by showing that frames really did go out. There is, however, a test that does the same job much more cheaply. Adding a one-line interface reading to the connection attempt brings group count to **6**, single-candidate group count to **4**, and the largest group drops from 3 to **2**. That is, a two-line configuration reading eliminates more at that same point than a 14400-line capture. Capture earns its keep when the remaining ambiguity is about the **existence of frames**; when the ambiguity is about configuration, reading the configuration is what is needed. Capture's second real function does not show up in the measurement: **timing**. The time between frames, retry intervals, and how much the return is delayed exist in no other test. This information eliminates no candidate, but it tells the duration; in an intermittent fault it is the only evidence, and that is the next lesson's subject. #### Capture's Boundary In this course, capture is a **diagnosis** tool, and no use of it beyond diagnosis is covered. Seizing someone else's traffic, spoofing identity, and evading monitoring are procedures written in **no lesson**; these are the Cybersecurity curriculum's subject, and even there they are treated not as a procedure but as a surface to be defended. The boundary itself follows from the measurement too. The line count needed for diagnosis is **six**; if the recording collects more than that, the extra adds nothing to the diagnosis. So the correct setup for capture is narrow: one interface, one target, one port, a counted number of frames. A wide setup is not just wasteful; it also records the content of unrelated flows, and that content belongs to no one's diagnosis. Three operating rules follow directly from this. Recording is bounded by **scope**: the filter is the flow under diagnosis. Recording is bounded by **duration**: frame count or duration is set in advance. The recording file is tied to a **retention policy**; the retention question measured for logs in the system administration course applies to the capture file exactly the same way, and a file is as much a burden as it is evidence. #### Candidate Set Sweep The sweep sets two tests side by side as the candidate set grows. In a two-candidate set, both eliminate 1 candidate and both finish the diagnosis; their difference is invisible. Once a fourth candidate is added, the paths split: the connection attempt eliminates 3 and stays at 1 left, capture eliminates 2 and climbs to **2** left. This gap holds through the eighth candidate; capture stays exactly **one** candidate behind at every step. What follows from this also bounds how much the measurement depends on the mock setup. Capture's weakness does not come from the line count or the chosen frame distribution; it comes from the **narrowness of the response space**. A three-valued response splits eight candidates into at best 3 groups, and even the smallest of those groups can hold more than one. As the candidate count grows, this narrowness grows with it: a one-candidate gap between the four-valued connection attempt and the three-valued capture widens further on a longer list. Every new candidate added to the list lowers the share the narrow-response test eliminates. #### Summary - Unfiltered capture produces **14400** lines, eliminates **6** candidates, and leaves **2** behind without finishing the diagnosis; a one-line connection attempt eliminates **7** and finishes it. - Eliminated candidates per line: **0.0004** for unfiltered capture, **7.0000** for the connection attempt. Growing the line count does not raise elimination power. - Capture alone never brings **any** fault down to a single candidate; the connection attempt brings **2** down to one, an interface reading also brings **2** down to one. - A filter lowers effort, not elimination power; a badly set filter can leave out the only evidence of the fault. - Capture earns its keep on ambiguity about the existence of frames and their timing; on ambiguity about configuration, a two-line reading eliminates more. - Capture is a diagnosis tool: it is bounded by scope, duration, and retention; use beyond diagnosis is not covered in this course. #### Next Step This whole lesson's measurement rested on a single assumption: capture was up and running before the fault began (**SG16**). Once that assumption is lifted, the numbers collapse, because an event with no recording looks as if it never happened. The next lesson takes up common fault patterns by their signatures and pays the third claim: while an intermittent fault lasts 84 seconds in a 600-second window, a recording starting at the 300th second sees half the events, one starting at the 500th sees less than a sixth; and a dropped packet's trace is zero lines while the recording is off. ### Common Fault Patterns Source: https://academia.sh/en/courses/linux-network-administration/common-fault-patterns The signatures of name-resolution, firewall, and path faults, and evidence vanishing on its own: an intermittent fault lasts 84 seconds in a 600-second window, a recording starting at second 300 catches 42/84, one starting at second 500 catches 13/84, and a dropped packet's trace stays at 0 lines while recording is off. The previous lesson's whole measurement rested on a single assumption: capture was up and running before the fault began. This lesson lifts that assumption and pays the course's third claim: **evidence does not wait.** A recording not taken while a fault is running cannot be taken afterward; an event with no recording never happened, as far as diagnosis is concerned. The lesson's first half finishes the practical question left over. The previous three lessons measured tests and their orders; a bundle of responses is now in hand. Now the view turns around: a given fault produces which bundle of responses, that is, what is its **signature**. Three patterns are taken up separately — name resolution, firewall, and path — because all three are common and all three produce symptoms that look alike. #### Three Patterns and Their Signatures A fault's **signature** is the tuple of responses the tests give for it (**SG17**). The signature is what diagnosis looks for: instead of narrowing the candidate set step by step, comparing the observed tuple against known tuples. The **name-resolution pattern** has the most distinct signature. The name query gets no response, but everything done by address works: echo returns, connection is established, the large packet gets through. The picture on the user's side is odd — the service works "sometimes," because when a cached record is used the query is not made at all. Name resolution's protocol itself is the Computer Networks curriculum's subject and is not opened here; the subject here is only the signature. The **firewall pattern's** signature is silence. When a packet is dropped, no response comes from the other side at all, and the connection attempt times out. This is exactly where it splits from being refused: refusal produces a response, dropping does not. The source of the silence this time is not the path but a rule in front of the target, and the path tests say sound. The **path pattern** is the sneakiest of the three. Small packets get through, large ones get lost. Echo says sound, connection is established, the first exchange works; it stops once it comes to a large data transfer. The user reports this as "slow" or "drops sometimes," and diagnosis starts being looked for somewhere related to speed. ```text # example dump , has not been run $ getent hosts # name-resolution signature $ nc -z -w 3 # refused , or timeout $ ping -c 3 -M do -s 1400 # path signature $ nft list ruleset # rules and counters read ``` ```python HEALTHY = {"interface_up": True, "address_set": True, "gateway_correct": True, "route_present": True, "resolver_responds": True, "service_listening": True, "firewall_passes": True, "path_ok": True} FAULT = { "interface-down": {"interface_up": False, "address_set": False}, "no-address": {"address_set": False}, "wrong-gateway": {"gateway_correct": False}, "missing-route": {"route_present": False}, "resolver-silent": {"resolver_responds": False}, "service-not-listening": {"service_listening": False}, "firewall-dropping": {"firewall_passes": False}, "large-packet-lost": {"path_ok": False}, } CANDIDATES = tuple(FAULT) def state(a=None): d = dict(HEALTHY) if a: d.update(FAULT[a]) return d def path(d): return (d["interface_up"] and d["address_set"] and d["gateway_correct"] and d["route_present"]) TEST = { "name-lookup": lambda d: "resolved" if d["resolver_responds"] else "unresolved", "listening": lambda d: "listening" if d["service_listening"] else "not-listening", "connection": lambda d: ("unreachable" if not path(d) else "timeout" if not d["firewall_passes"] else "refused" if not d["service_listening"] else "established"), "large-packet": lambda d: ("passes" if path(d) and d["path_ok"] else "blocked"), } UPPER = list(TEST) # SG17: signature = tuple of test responses def signature(a, kit=UPPER): return tuple(TEST[s](state(a)) for s in kit) def groups(kit, candidates=CANDIDATES): o = {} for a in candidates: o.setdefault(signature(a, kit), []).append(a) return o print("fault name-lookup listening connection large-packet") for a in CANDIDATES: i = signature(a) print(f" {a:26s}{i[0]:14s}{i[1]:15s}{i[2]:13s}{i[3]}") o = groups(UPPER) print("distinct signatures:", len(o), "| unsplit groups:", len([v for v in o.values() if len(v) > 1]), "| largest group:", max(len(v) for v in o.values())) print() print("a test's elimination power changes with its response") for s in TEST: for response, fits in sorted(groups([s]).items()): print(f" {s:13s} {response[0]:12s} eliminated {len(CANDIDATES) - len(fits)} left {len(fits)}") print() def intermittent(window=600, period=45, duration=6): # SG18 return [t for t in range(window) if t % period < duration] def recording_catches(events, start): # SG19 y = [t for t in events if t >= start] return len(y), round(len(y) / len(events), 4) def dropped_packet_trace(recording_on): # SG20 return 128 if recording_on else 0 EVENTS = intermittent() print("intermittent fault: window 600 s , event", len(EVENTS), "seconds ,", len(EVENTS) // 6, "attacks x 6 seconds") for start in (0, 100, 300, 500): y, ratio = recording_catches(EVENTS, start) print(f" recording starts at second {start:3d} -> caught {y:3d} / {len(EVENTS)} = {ratio}") print("dropped packet trace: recording off ->", dropped_packet_trace(False), "lines , recording on ->", dropped_packet_trace(True), "lines") print() print("candidate set sweep: name-lookup test's two responses") kit = ["resolver-silent"] for a in CANDIDATES: if a not in kit: kit.append(a) o = groups(["name-lookup"], tuple(kit)) negative = len(o[("unresolved",)]) positive = len(o.get(("resolved",), [])) print(f" candidate {len(kit)} 'unresolved' eliminated {len(kit) - negative} left {negative}" f" 'resolved' eliminated {len(kit) - positive} left {positive}") ``` ``` fault name-lookup listening connection large-packet interface-down resolved listening unreachable blocked no-address resolved listening unreachable blocked wrong-gateway resolved listening unreachable blocked missing-route resolved listening unreachable blocked resolver-silent unresolved listening established passes service-not-listening resolved not-listening refused passes firewall-dropping resolved listening timeout passes large-packet-lost resolved listening established blocked distinct signatures: 5 | unsplit groups: 1 | largest group: 4 a test's elimination power changes with its response name-lookup resolved eliminated 1 left 7 name-lookup unresolved eliminated 7 left 1 listening listening eliminated 1 left 7 listening not-listening eliminated 7 left 1 connection established eliminated 6 left 2 connection refused eliminated 7 left 1 connection timeout eliminated 7 left 1 connection unreachable eliminated 4 left 4 large-packet blocked eliminated 3 left 5 large-packet passes eliminated 5 left 3 intermittent fault: window 600 s , event 84 seconds , 14 attacks x 6 seconds recording starts at second 0 -> caught 84 / 84 = 1.0 recording starts at second 100 -> caught 66 / 84 = 0.7857 recording starts at second 300 -> caught 42 / 84 = 0.5 recording starts at second 500 -> caught 13 / 84 = 0.1548 dropped packet trace: recording off -> 0 lines , recording on -> 128 lines candidate set sweep: name-lookup test's two responses candidate 2 'unresolved' eliminated 1 left 1 'resolved' eliminated 1 left 1 candidate 3 'unresolved' eliminated 2 left 1 'resolved' eliminated 1 left 2 candidate 4 'unresolved' eliminated 3 left 1 'resolved' eliminated 1 left 3 candidate 5 'unresolved' eliminated 4 left 1 'resolved' eliminated 1 left 4 candidate 6 'unresolved' eliminated 5 left 1 'resolved' eliminated 1 left 5 candidate 7 'unresolved' eliminated 6 left 1 'resolved' eliminated 1 left 6 candidate 8 'unresolved' eliminated 7 left 1 'resolved' eliminated 1 left 7 ``` #### Oracle, Test, Eliminated Candidate Three numbers sit side by side. **Oracle:** the real fault is `resolver-silent`. **Test:** the name query returns `unresolved`. **Eliminated candidate:** this response eliminates **7** candidates and leaves **1** behind; the signature is unique and the diagnosis finishes. The same test, when the oracle is `firewall-dropping`, says `resolved` and eliminates only **1** candidate. That is, the name-resolution test was the weakest tool in one lesson and here it is the strongest. What changed is not the tool, it is the **response**. The signature table splits eight faults into **5** distinct signatures using four upper-layer tests. The one unsplit group is the lower layer's four: interface, address, gateway, and route faults all produce the **same** tuple across these four tests. This is the reverse of the result measured in the second lesson, where reachability tests left the upper four in a single group. The result is symmetric: **every test kit blinds itself to what sits outside its own layer**, and the split completes only once tests are drawn from both layers. #### A Negative Response Eliminates, a Positive One Does Not The second table is this lesson's most generalizable finding. A test's elimination power is not a fixed number; it changes with its response, and the gap is large. Name lookup eliminates **7** candidates when it says `unresolved`, **1** when it says `resolved`. The listening list eliminates **7** when it says `not-listening`, **1** when it says `listening`. The connection attempt's four responses eliminate 6, 7, 4, and 7 candidates respectively. The reason for the pattern sits in the mock setup's structure: because each fault breaks a single field of the healthy state, that field coming up broken points to that one fault; coming up healthy is consistent with the remaining seven. The practical takeaway is direct: **good news is little information.** The line "name resolution is working" advances the diagnosis almost not at all; the line "name resolution is not working" finishes it. This also changes test selection. A test's expected elimination power depends on how its responses are distributed over the candidate set. A test with a balanced response distribution does similar work in every case; a test with a heavily skewed distribution rarely does much and mostly does little. The connection attempt's four responses are the most balanced in this respect, eliminating at least 4 candidates even in the worst case. #### Intermittent Fault: Evidence Does Not Wait Up to here, every test was run while the fault was active. There is no such guarantee for an intermittent fault. In the mock setup, the fault lasts 6 seconds every 45 seconds; a 600-second window holds **14 attacks** and a total of **84 seconds** of fault (**SG18**). The recording starts at one moment and runs to the window's end (**SG19**). The numbers are plain. If the recording starts at the zeroth second, **84** of 84 events are seen. If it starts at the hundredth second, **66** (0.7857). At the three-hundredth second, **42** (0.5000). At the five-hundredth second, **13** (0.1548). Starting halfway through the recording window loses exactly half of the events; starting five-sixths through loses more than five-sixths — eighty-four percent. Two conclusions follow. First, **a late-starting recording measures the wrong frequency.** An operator looking at a recording that started at the five-hundredth second sees thirteen seconds of fault and calls the event "occasional"; the real duration is eighty-four seconds. What is wrong is not the observation, it is the assumption about how much of a window that observation represents. The damage an intermittent fault does to the signature method is even heavier. A signature is correct only when taken while the fault is running. In the mock setup, the fault occupies 84 of the window's 600 seconds, that is, **0.1400** of it; that is the share of a single test done at a random moment landing on the fault. In the remaining eighty-six percent, the test returns `healthy`, and by the rule measured in the previous section, that response **eliminates** the real cause. The candidate eliminated from the set is the correct one, and this shows no sign anywhere along the way: every test is consistent, the signature is a healthy system's signature, and the service still does not work. For this reason, the right move against an intermittent fault is to **repeat** the test, but the warning written in the previous lesson loses its force here: there, repeating the same test eliminated no new candidate, because the system's state was fixed. If the state changes over time, repetition produces new information, and the information produced is no longer a single response, it is a **response distribution**. A test that comes back healthy in two of three tries and broken in one says something no single response could say. Second, the recording's **start moment is a diagnostic decision**, and it has to be made before the fault is reported. Opening a recording once a fault is heard makes everything from that moment on visible; none of the fourteen prior attacks come back. Log rotation was measured, in the system administration course, as **erasing** evidence; here evidence is not erased, it is never **written** in the first place. The two share the same conclusion, and the same remedy: the decision has to be made before the fault occurs. #### The Trace a Dropped Packet Leaves The last number is the shortest one. When the firewall drops a packet, the default behavior is to leave no trace: with the recording off, the dropped packet's trace is **0 lines**; with it on, **128 lines** (**SG20**). Both cases are the same fault; the only thing that changed is a setting. This is the third claim in its harshest form. Silent dropping is diagnosis's most expensive fault, because no test sees it directly; it is only **inferred** from the timeout signature. With tracing on, the same fault is read in a single line. That is, the 128-line gap stands in for what would otherwise be a six-test inference. The trace's cost has to be written up too. If every dropped packet produces a line, a heavy drop rate turns the recording itself into a load, speeding up log rotation and erasing other evidence. The right setup is not continuous tracing but **bounded** tracing: for a specific rule, for a set duration, with a capped count. This is a decision made before the fault; made during the fault, it comes too late. The same arithmetic combined with an intermittent fault produces the worst case: a silently dropped packet, if the fault is also intermittent, shows up in neither the trace nor the signature. Such a fault is read only from a repeated test's response distribution, and that distribution never forms unless someone repeats the test. #### Candidate Set Sweep The sweep shows how the gap between a negative and a positive response grows with the set. In a two-candidate set, both responses eliminate 1 candidate; there is no difference. Once the set grows to eight, `unresolved` eliminates **7** candidates, `resolved` still eliminates **1**. The negative response's power grows linearly with the set, the positive one's stays **fixed**. This is a result that does not depend on the mock setup, and its reason is structural: because only a single fault breaks a given field, that field coming up broken points to the one fault regardless of candidate count; coming up healthy is consistent with the remaining seven. Adding a new fault to the candidate list lowers the positive response's value and raises the negative response's. Widening the list is therefore not "being more careful" — it changes the measurement itself, and it changes which test will do the job. #### Summary - A fault's signature is the tuple of responses the tests give for it; four upper-layer tests split eight faults into **5** signatures, and the lower layer's four faults stay in one group. - A test's elimination power changes with its response: name lookup eliminates **7** when it says `unresolved`, **1** when it says `resolved`. Good news is little information. - An intermittent fault lasts **84 seconds** in a 600-second window; a recording starting at second 300 catches **42/84** (0.5000), one starting at second 500 catches **13/84** (0.1548). - A late-starting recording does not just see less, it measures the **wrong frequency**; a thirteen-second observation produces the conclusion "occasional." - A dropped packet's trace is **0 lines** with recording off, **128** with it on; the fault is the same fault, visibility is a setting, and that setting has to be turned on before the fault. - A negative response's elimination power grows with the candidate set, a positive response's stays fixed at **1**. #### Next Step This topic showed how much of diagnosis depends on decisions made **beforehand**: the recording being on, tracing being kept bounded, the candidate list being written down. The same question comes back harder on the access side. Connecting to a server remotely is itself a configuration decision, and if that decision is wrong, the channel that would do the diagnosing closes too. The next topic covers secure shell access, key-based authentication, and local hardening; then it moves to the packet-filtering framework and measures the growth of the externally exposed surface by counting it. ## Access and Filtering Key-based remote access, tunneling and the jump server, the packet-filtering framework, the default-deny policy, and counting the surface address translation opens. ### Secure Shell Source: https://academia.sh/en/courses/linux-network-administration/secure-shell Key-based authentication and server hardening are established; the nine candidate causes of a session failing to open are eliminated with seven tests, and a forty-six-line verbose client output eliminates five candidates against a three-line server log's seven. The previous topic chased a fault's trace and showed that the trace is often not left for someone looking late: an intermittent fault begins before the recording starts, a dropped packet writes no line by default. This topic asks the same question of access itself. When the way to connect to a server closes, a single line sits in front of you, and that line usually says "permission denied." The question to ask is how many candidate causes that line eliminates. The course's rule does not change — a test's number is not the answer it gives, but **what it eliminates**. In this lesson, even when the tests are put together, a pair will remain that they cannot eliminate, and what separates that pair is not a tool but a setting made **before** the fault. #### The Two Ends of the Key Pair Two separate authentications stand stacked in a secure shell connection, and confusing them breaks the diagnosis. In the first, **the client authenticates the server**: the server's **host key** is recorded on the first connection and compared against the record on later connections. In the second, **the server authenticates the user**: the user's **private key** stays on the client, and its counterpart, the **public key**, is found in the authorized keys file on the server. The key pair is generated on the client, and the private key never leaves it; only the public key is copied to the server. This asymmetry is the basis of hardening: even if the server is compromised, what sits there is not enough to produce the user's identity. **No real key, fingerprint, or password is written in this lesson;** wherever a key string would be, a placeholder in square brackets is used. Server-side hardening consists of a few settings. The dump below is not executed; it is an example written to show the settings' names and the values they take: ```text # example dump, not executed PermitRootLogin no PasswordAuthentication no PubkeyAuthentication yes AuthorizedKeysFile .ssh/authorized_keys LogLevel VERBOSE ``` Each of these lines changes the candidate-cause set, and this is the measurement's real subject. Turning off password authentication removes every password-related candidate from the list and puts one in its place: no user without a key is left a fallback method. Fixing the location of the authorized keys file removes the candidate of the file being looked for in the wrong place. Raising the log's verbosity level removes no candidate, **but it increases distinguishability**, and that is exactly what this lesson's final measure counts. There is one more setting, and it is often overlooked: if the permissions of the authorized keys file and the directory containing it are too open, the server **will not read** the file. The file is in place, its content is correct, the key would match; the server still refuses. This behavior adds a separate candidate cause to the list, and it is the candidate that produces the most frustrating result in the measurement. #### The Public Key Crossing to the Other Side Once the key pair is generated, one task remains: **appending** the public key to the correct user's authorized keys file on the server. This task produces more candidate causes than it looks like it should, because it can go wrong in four separate places. The key can be appended to the wrong user's file; the file can be correct but merge with the previous line because a line ending is missing; if the directory does not exist on the server at all, the append can silently fall to a different path; a space per line entered during copying can invalidate the key. What the four share is that the operation's output comes back **successful**. For this reason, the key being placed is not verified on its own. Verification is done not by the operation's output but by **a new session attempt**, and the attempt is made without closing the current session. This detail will be repeated exactly the same way in rule writing in later lessons: what verifies a change is not the value returned by the command that made the change, but an attempt made through a separate channel. The key's lifetime, its regular rotation, being stored in a central vault, and being signed with a certificate are not this course's subject; these are measured in the key management lessons of the **Cybersecurity** curriculum. What is measured here is only how much a single machine's local configuration makes distinguishable in the face of a fault. #### Nine Candidate Causes The setup is this: the user is trying to connect and the session does not open. There are nine candidate causes, each breaking a single field of the healthy state. The shared reference's eight network faults are gathered here into a single candidate — the network being unreachable; this topic measures the layer above the network. - **EF1** — The candidate-cause list has **nine** items and is exhaustive; the setup never goes outside these nine states. - **EF2** — The oracle is chosen as `public-key-unauthorized`: the user has a key, and it has no counterpart in the server's authorized keys file. - **EF3** — Seven tests are defined. Four can be done **from outside**, without opening a session; two require running **on the server**, and one is the server log's basic level. - **EF4** — Each test is given a **line count**. This number is part of the setup, not a measured value; its purpose is to place output volume and elimination power side by side. - **EF5** — The verbose client output is counted as **46** lines, the verbose server log **3**, the basic server log **1**, the listening list **12**, the local key check **4**, the method list **9**, the default client message **1**. - **EF6** — Verbose logging is assumed to have been turned on **before** the fault; turning it on afterward does not bring back a past attempt. - **EF7** — The breadth of the key permissions and the public key being unauthorized fall into **the same failure branch** on the server; this reflects real behavior, not the setup. - **EF8** — The candidate set sweep is done with three lists: eight, nine, and ten candidates. - **EF9** — `method-disabled`, added to the ten-candidate list, is the case of the server rejecting the key method, and it gives **the same answer pattern** as `public-key-unauthorized`. - **EF10** — All counts are exhaustive; there is no randomness or seed. ```python """Secure shell session does not open: nine candidate causes, seven tests. Oracle = the real cause (we chose the setup). What is measured is how many candidates each test eliminates and how many remain. """ CANDIDATE = ("network-unreachable", "service-not-listening", "host-key-changed", "private-key-missing", "public-key-unauthorized", "key-permissions-too-open", "user-missing", "account-locked", "password-method-disabled") HEALTHY = {"transport_established": True, "service_listening": True, "host_key_matches": True, "client_offers_key": True, "server_accepts": True, "account_valid": True, "password_open": True} FAULT = { "network-unreachable": {"transport_established": False}, "service-not-listening": {"transport_established": False, "service_listening": False}, "host-key-changed": {"host_key_matches": False}, "private-key-missing": {"client_offers_key": False, "server_accepts": False}, "public-key-unauthorized": {"server_accepts": False}, "key-permissions-too-open": {"server_accepts": False}, "user-missing": {"server_accepts": False, "account_valid": False}, "account-locked": {"server_accepts": False, "account_valid": False}, "password-method-disabled": {"password_open": False}, } LINES = {"client-message": 1, "verbose-client": 46, "local-key": 4, "method-list": 9, "listening": 12, "log-basic": 1, "log-verbose": 3} def state(a=None): d = dict(HEALTHY) if a: d.update(FAULT[a]) return d def test_client(d): if not d["transport_established"]: return "no-connection" if not d["host_key_matches"]: return "host-key-warning" return "session-opened" if d["server_accepts"] else "permission-denied" def test_verbose_client(d): if not d["transport_established"]: return "transport-failed" if not d["host_key_matches"]: return "host-key-mismatch" if not d["client_offers_key"]: return "key-not-offered" return "session-opened" if d["server_accepts"] else "key-rejected" def test_log_basic(d): if not d["transport_established"]: return "no-line" return "accepted" if d["server_accepts"] else "failed" def test_log_verbose(d): if not d["transport_established"]: return "no-line" if not d["account_valid"]: return "invalid-account" if not d["client_offers_key"]: return "method-not-offered" return "accepted" if d["server_accepts"] else "key-mismatch" TEST = { "client-message": test_client, "verbose-client": test_verbose_client, "local-key": lambda d: ("key-present" if d["client_offers_key"] else "key-absent"), "method-list": lambda d: ("password-open" if d["password_open"] else "password-closed"), "listening": lambda d: "listening" if d["service_listening"] else "not-listening", "log-basic": test_log_basic, "log-verbose": test_log_verbose, } FROM_OUTSIDE = ["client-message", "verbose-client", "local-key", "method-list"] ON_SERVER = ["listening", "log-verbose"] ORACLE = "public-key-unauthorized" def eliminate(candidates, s, answer): return tuple(a for a in candidates if TEST[s](state(a)) == answer) def group(tests, candidates=CANDIDATE): o = {} for a in candidates: o.setdefault(tuple(TEST[s](state(a)) for s in tests), []).append(a) return [sorted(v) for v in o.values() if len(v) > 1] print("candidate causes:", len(CANDIDATE), "| tests:", len(TEST), "| oracle:", ORACLE) print() print("test lines distinct answers eliminated remaining per line") for s in TEST: remaining = len(eliminate(CANDIDATE, s, TEST[s](state(ORACLE)))) el = len(CANDIDATE) - remaining print(f" {s:18s} {LINES[s]:5d} {len({TEST[s](state(a)) for a in CANDIDATE}):11d}" f" {el:7d} {remaining:6d} {el / LINES[s]:13.4f}") print() print("tool set lines indistinguishable groups largest group") for ad, k in (("client message alone", ["client-message"]), ("verbose client alone", ["verbose-client"]), ("4 accessible from outside", FROM_OUTSIDE), ("outside 4 + basic log", FROM_OUTSIDE + ["log-basic"]), ("2 running on server", ON_SERVER), ("seven tests", list(TEST))): o = group(k) print(f" {ad:28s} {sum(LINES[s] for s in k):5d} {len(o):17d}" f" {max((len(x) for x in o), default=0):14d}") print() print("candidate set sweep (same oracle)") FAULT["method-disabled"] = {"server_accepts": False} for ad, k in (("8 candidates: account-locked removed", tuple(a for a in CANDIDATE if a != "account-locked")), ("9 candidates: base list", CANDIDATE), ("10 candidates: method-disabled added", CANDIDATE + ("method-disabled",))): remaining = len(eliminate(k, "log-verbose", TEST["log-verbose"](state(ORACLE)))) o = group(list(TEST), k) print(f" {ad:35s} verbose log eliminated {len(k) - remaining:2d}" f" remaining {remaining:2d} | indistinguishable group {len(o)}" f" largest {max((len(x) for x in o), default=0)}") print() print("indistinguishable groups (seven tests):", group(list(TEST))) ``` ``` candidate causes: 9 | tests: 7 | oracle: public-key-unauthorized test lines distinct answers eliminated remaining per line client-message 1 4 4 5 4.0000 verbose-client 46 5 5 4 0.1087 local-key 4 2 1 8 0.2500 method-list 9 2 1 8 0.1111 listening 12 2 1 8 0.0833 log-basic 1 3 4 5 4.0000 log-verbose 3 5 7 2 2.3333 tool set lines indistinguishable groups largest group client message alone 1 2 5 verbose client alone 46 2 4 4 accessible from outside 60 2 4 outside 4 + basic log 61 2 4 2 running on server 15 3 2 seven tests 76 2 2 candidate set sweep (same oracle) 8 candidates: account-locked removed verbose log eliminated 6 remaining 2 | indistinguishable group 1 largest 2 9 candidates: base list verbose log eliminated 7 remaining 2 | indistinguishable group 2 largest 2 10 candidates: method-disabled added verbose log eliminated 7 remaining 3 | indistinguishable group 2 largest 3 indistinguishable groups (seven tests): [['key-permissions-too-open', 'public-key-unauthorized'], ['account-locked', 'user-missing']] ``` #### Three Numbers **Oracle:** the real cause is `public-key-unauthorized` — the key is offered, the server does not recognize it. **Test:** the default client message says "permission denied" in a single line; the verbose client output prints forty-six lines and says "key rejected"; the verbose server log says "key mismatch" in three lines. **Candidates eliminated:** the default message **4**, the verbose client **5**, the verbose server log **7**. The comparison in the middle is this lesson's form of the course's second claim. The verbose client output produces **fifteen times** as many lines as the server log and eliminates **two fewer** candidates. The per-line elimination ratio is 0.1087 against 2.3333 — a factor of twenty-one apart. The reason is structural: the verbose client output details **the client side of the handshake**, while every one of the unseparated candidates is on the server side. Raising the verbosity level enlarges the side that is visible; it does not open up the side that is not. The same table's weakest row is `listening`. It prints twelve lines and eliminates only **one** candidate. In the shared reference, the listening list's elimination power was also one; on the access layer, it is one as well. Seeing that the service is listening does not show that you can connect, and the distance between these two sentences is this entire course. #### The Pair That Cannot Be Separated The bottom table says something harsher. Even when all seven tests are used together, **two groups** remain unseparated, and both have two members: `public-key-unauthorized` with `key-permissions-too-open`, `account-locked` with `user-missing`. Seventy-six lines of output are produced, and these four states cannot be split two and two. The first pair ties back to the lesson's hardening section. If the authorized keys file's permissions are too open, the server does not read the file and refuses; it also refuses if the public key is not in the file. From the server's point of view, both are in the "the offered key is not acceptable" branch. What will show the difference between them is not a test, it is **what the server writes on that branch** — and what it writes depends on the log's verbosity level. This is the access layer's form of the course's third claim. Kept at the basic level, the log gives three distinct answers and eliminates four candidates; at the verbose level it gives five answers and eliminates seven. The difference is hidden in a three-line output. But the level must have been raised **before** the fault: detail turned on afterward does not bring back a past attempt. **The evidence does not wait.** One more row in the table reinforces this. The four tests reachable from outside produce sixty lines and leave the largest unseparated group at **four members**. The two tests running on the server produce fifteen lines and leave the largest group at **two members**. All of the elimination power sits on the server; the problem is that getting onto the server is closed. The answer to this is leaving a second path alongside secure shell: a console that can be logged, or a separate management channel. Without that path, diagnosis stays limited to the four tests reachable from outside, and the four-member group stays exactly as it is. #### Candidate Set Sweep There is no second seed in this course, because there is no randomness. In its place, the candidate list is changed. The table below puts three lists side by side, and what needs reading is not the eliminated column but the **remaining** column. When `account-locked` is removed from the list, the number of candidates the verbose log eliminates drops from seven to six; what remains is two, and the indistinguishable group count drops from two to one. When `method-disabled` is added to the list, the eliminated count **does not change** — still seven — but what remains rises from two to **three**, and the largest indistinguishable group widens from two members to three. Same test, same answer, same elimination count; the diagnosis got worse. The rule that follows is this: **elimination count alone is not a measure.** As the candidate list grows, the same test eliminates the same number of candidates and leaves more behind. What needs to be measured is not elimination but **remaining ambiguity**. This is the second half of the course's rule, and it will come up again and again in the following lessons as the jump server, chains, and address translation grow the candidate list. #### Summary - Two authentications stand stacked in a secure shell connection: the client authenticates the server with the host key, the server authenticates the user with the public key. The private key never leaves the client. - Hardening settings change the candidate-cause list: turning off password authentication removes password-related candidates, the authorized keys file's permissions add a new candidate. - Across nine candidate causes and seven tests, the eliminated counts are not equal: the verbose server log eliminates **7** candidates in three lines, the verbose client output **5** in forty-six lines, the listening list **1** in twelve lines. - When the seven tests are used together, **two groups** cannot be separated; what does the separating is not a tool but the log's verbosity level, raised before the fault. The basic level eliminates **4**, the verbose level **7** candidates. - When one item is added to the candidate list, the eliminated count stays fixed while what remains rises from **2 to 3**; what needs to be measured is not elimination but remaining ambiguity. #### Next Step This lesson's sharpest result was that the elimination power sits on the server, and getting onto the server is closed. The four tests reachable from outside leave a four-member group exactly as it is. The next lesson takes up the usual solution to this blockage: going to a machine that cannot be reached directly, by way of another machine that can be reached. A jump server and a tunnel split the path in two, and they split the diagnosis in two as well — the fault can now be on either of two machines. The question to measure is how many of this expanded candidate set a single end-to-end attempt eliminates. ### Tunneling and the Jump Server Source: https://academia.sh/en/courses/linux-network-administration/tunneling-and-the-jump-server Splitting the path in two brings the candidate-cause set up to ten; a single end-to-end attempt eliminates eight and leaves three indistinguishable groups between two hops, a test done from the hop eliminates nine, and the tunnel entry point's bind address raises indirect access from zero to seventy-two. The previous lesson's sharpest result was this: all the elimination power sits on the server, and the way onto the server is closed. The four tests that can be done from outside left a four-member group unseparated. The usual solution to this blockage is going to a machine that cannot be reached directly, by way of another machine that can be reached. The solution comes with a cost, and this lesson counts that cost. When the path splits in two, the candidate-cause set splits in two as well: the fault can now be on either of the two machines, on either of the two legs in between, or on the client itself. How much of this expanded set a single end-to-end attempt eliminates is the question. #### The Path's Two Stops A **jump host** is the machine that stands as the sole gate between two networks. The client connects to it, and it connects to the target. There are two arrangements, and they produce different candidate causes. In the first, a **local port** is opened on the client machine, and traffic arriving at that port is carried through the encrypted session and forwarded to the target. This is called a **tunnel**. In the second, the jump host is only a carrier: the session opens directly on the target, and no shell runs on the jump host. The dump below is not executed; it is meant to show the syntax of the two forms: ```text # example dump, not executed # local forwarding: the mouth opens on the client, traffic goes to the target via the jump ssh -L 127.0.0.1::: @ # using as a hop: the session opens on the target, the jump only carries ssh -J @ @ ``` The detail shared by both forms decides half the diagnosis: the `` name is **resolved on the jump host**, not on the client. These two forms work only if the target's name is defined in the remote network's resolver. If the same name is written as an alias in the client's own configuration, the client tries to resolve it and cannot. The resulting error message points at name resolution, even though the name resolves perfectly well on the server. **Which side** a name resolves on is this topic's own particular candidate cause. How a jump host narrows the number of directly reachable machines was counted in the remote-access hardening lesson of the **DevOps and Platform Engineering** curriculum, and its placement between trust zones in the perimeter defense topic of the **Cybersecurity** curriculum. What is measured here is different: what the same arrangement does to **diagnosis**. #### Where the Tunnel's Mouth Binds The tunnel opens a port on the client machine, and that port's **bind address** is a choice. Bound to the loopback address, only processes on that machine can use the tunnel. Bound to all interfaces, every host on the client's network segment can use it too — and at the other end of the tunnel sits a target those hosts cannot reach directly. This is a countable growth of exposed surface. In the setup, **24** other hosts sit on the client's segment, and **3** tunnels are open. With tunnels bound to loopback, the number of other hosts that can reach the tunnel's mouth is **0**, and the number of (host, target service) pairs that become indirectly reachable is **0**. Bound to all interfaces, these two numbers become **24** and **72**. The difference is a single configuration field, and a tunnel written without knowing its default opens a path thought closed to twenty-four hosts. #### Ten Candidate Causes The setup is this: the client is trying to reach a service on the target through the jump host and cannot. There are ten candidate causes, each breaking a single field of the healthy state. - **EF11** — The candidate-cause list has **ten** items and is exhaustive. Two are on the client, two on the first leg, one in the jump host's configuration, four on the target, one in name resolution's location. - **EF12** — The oracle is chosen as `target-firewall-dropping`: the target's filter silently drops the packet. - **EF13** — Six tests are defined. Three can be done on the client, two require opening a session on the jump host, one compares the two sides. - **EF14** — Tests that run on the jump host **cannot be done** if the first leg is broken, and they report this with a separate answer; this is a dependency and is part of the measurement. - **EF15** — The end-to-end attempt covers opening the tunnel and a session attempt through it together; for this reason it can also see the authentication result on the target. - **EF16** — The layered order proceeds from client to target; the second order puts the end-to-end attempt first. Both can be applied without knowing the oracle. - **EF17** — The tunnel entry point measurement assumes **24** other hosts and **3** open tunnels in the segment. - **EF18** — The candidate set sweep is done with nine-, ten-, and eleven-candidate lists. - **EF19** — `jump-session-restricted`, added to the eleven-candidate list, is the jump host blocking forwarding at the session level, and it gives the same answer as `forwarding-disabled`. - **EF20** — All counts are exhaustive; there is no randomness or seed. ```python """Two-hop path: client -> jump host -> target. Ten candidates, six tests.""" CANDIDATE = ("local-port-busy", "local-address-wrong", "jump-unreachable", "jump-auth-refused", "forwarding-disabled", "target-unreachable", "target-firewall-dropping", "target-not-listening", "target-auth-refused", "name-resolves-remote-only") HEALTHY = {"local_bound": True, "local_address_correct": True, "jump_reachable": True, "jump_auth": True, "forwarding_allowed": True, "target_reachable": True, "target_firewall_allows": True, "target_listening": True, "target_auth": True, "name_resolves_locally": True} FAULT = { "local-port-busy": {"local_bound": False}, "local-address-wrong": {"local_address_correct": False}, "jump-unreachable": {"jump_reachable": False}, "jump-auth-refused": {"jump_auth": False}, "forwarding-disabled": {"forwarding_allowed": False}, "target-unreachable": {"target_reachable": False}, "target-firewall-dropping": {"target_firewall_allows": False}, "target-not-listening": {"target_listening": False}, "target-auth-refused": {"target_auth": False}, "name-resolves-remote-only": {"name_resolves_locally": False}, } def state(a=None): d = dict(HEALTHY) if a: d.update(FAULT[a]) return d def test_end_to_end(d): if not d["local_bound"] or not d["local_address_correct"]: return "local-connection-refused" if not d["name_resolves_locally"]: return "name-unresolved" if not d["jump_reachable"] or not d["jump_auth"]: return "hop-not-opened" if not d["forwarding_allowed"]: return "channel-forbidden" if not d["target_reachable"] or not d["target_firewall_allows"]: return "channel-timeout" if not d["target_listening"]: return "channel-refused" return "established" if d["target_auth"] else "permission-denied-at-target" def test_jump_session(d): if not d["jump_reachable"]: return "unreachable" return "opened" if d["jump_auth"] else "permission-denied" def test_local_listener(d): if not d["local_bound"]: return "not-bound" return "expected-address" if d["local_address_correct"] else "different-address" def test_jump_to_target(d): if not d["jump_reachable"] or not d["jump_auth"]: return "could-not-be-done" if not d["target_reachable"]: return "unreachable" if not d["target_firewall_allows"]: return "timeout" return "established" if d["target_listening"] else "refused" def test_forwarding_setting(d): if not d["jump_reachable"] or not d["jump_auth"]: return "unreadable" return "allowed" if d["forwarding_allowed"] else "disabled" TEST = {"end-to-end": test_end_to_end, "local-listener": test_local_listener, "jump-session": test_jump_session, "forwarding-setting": test_forwarding_setting, "jump-to-target": test_jump_to_target, "name-location": lambda d: ("both-sides" if d["name_resolves_locally"] else "remote-only")} LAYERED = ["local-listener", "name-location", "jump-session", "forwarding-setting", "jump-to-target", "end-to-end"] END_TO_END_FIRST = ["end-to-end"] + LAYERED[:-1] def eliminate(candidates, s, answer): return tuple(a for a in candidates if TEST[s](state(a)) == answer) def group(tests, candidates=CANDIDATE): o = {} for a in candidates: o.setdefault(tuple(TEST[s](state(a)) for s in tests), []).append(a) return [sorted(v) for v in o.values() if len(v) > 1] def order_steps(order, real_fault, candidates=CANDIDATE): remaining, step = tuple(candidates), 0 for s in order: if len(remaining) <= 1: break step += 1 remaining = eliminate(remaining, s, TEST[s](state(real_fault))) return step, len(remaining) ORACLE = "target-firewall-dropping" print("candidate causes:", len(CANDIDATE), "| tests:", len(TEST), "| oracle:", ORACLE) print() print("test distinct answers worst-case remaining eliminated remaining") for s in TEST: o = {} for a in CANDIDATE: o.setdefault(TEST[s](state(a)), []).append(a) remaining = len(eliminate(CANDIDATE, s, TEST[s](state(ORACLE)))) print(f" {s:18s} {len(o):11d} {max(len(v) for v in o.values()):14d}" f" {len(CANDIDATE) - remaining:7d} {remaining:6d}") print() print("fault layered end-to-end-first end-to-end alone remaining") tk = tu = 0 for a in CANDIDATE: k, _ = order_steps(LAYERED, a) u, _ = order_steps(END_TO_END_FIRST, a) _, y = order_steps(["end-to-end"], a) tk += k tu += u print(f" {a:26s} {k:8d} {u:11d} {y:22d}") print(f" {'total':26s} {tk:8d} {tu:11d}") print() print("tool set indistinguishable group") for ad, k in (("end-to-end alone", ["end-to-end"]), ("end-to-end + jump session", ["end-to-end", "jump-session"]), ("end-to-end + jump-to-target", ["end-to-end", "jump-to-target"]), ("three tests running locally", ["end-to-end", "local-listener", "name-location"]), ("six tests", list(TEST))): o = group(k) print(f" {ad:35s} {len(o):8d}", o if o else "") print() NEIGHBORS, TUNNELS = 24, 3 print(f"the tunnel entry point's bind address ({NEIGHBORS} other hosts in the subnet, {TUNNELS} tunnels open)") for ad, reachers in (("loopback", 0), ("all interfaces", NEIGHBORS)): print(f" {ad:14s} other hosts that can reach the tunnel entry {reachers:3d}" f" | indirect (host, target service) pairs {reachers * TUNNELS:3d}") print() print("candidate set sweep") FAULT["jump-session-restricted"] = {"forwarding_allowed": False} for ad, k in (("9 candidates: target-auth-refused removed", tuple(a for a in CANDIDATE if a != "target-auth-refused")), ("10 candidates: base list", CANDIDATE), ("11 candidates: jump-session-restricted added", CANDIDATE + ("jump-session-restricted",))): remaining = len(eliminate(k, "end-to-end", TEST["end-to-end"](state(ORACLE)))) o = group(list(TEST), k) print(f" {ad:42s} end-to-end eliminated {len(k) - remaining:2d} remaining {remaining:2d}" f" | indistinguishable group {len(o)}" f" largest {max((len(x) for x in o), default=0)}") ``` ``` candidate causes: 10 | tests: 6 | oracle: target-firewall-dropping test distinct answers worst-case remaining eliminated remaining end-to-end 7 2 8 2 local-listener 3 8 2 8 jump-session 3 8 2 8 forwarding-setting 3 7 3 7 jump-to-target 5 5 9 1 name-location 2 9 1 9 fault layered end-to-end-first end-to-end alone remaining local-port-busy 1 2 2 local-address-wrong 1 2 2 jump-unreachable 3 4 2 jump-auth-refused 3 4 2 forwarding-disabled 4 1 1 target-unreachable 5 6 2 target-firewall-dropping 5 6 2 target-not-listening 5 1 1 target-auth-refused 5 1 1 name-resolves-remote-only 2 1 1 total 34 28 tool set indistinguishable group end-to-end alone 3 [['local-address-wrong', 'local-port-busy'], ['jump-auth-refused', 'jump-unreachable'], ['target-firewall-dropping', 'target-unreachable']] end-to-end + jump session 2 [['local-address-wrong', 'local-port-busy'], ['target-firewall-dropping', 'target-unreachable']] end-to-end + jump-to-target 2 [['local-address-wrong', 'local-port-busy'], ['jump-auth-refused', 'jump-unreachable']] three tests running locally 2 [['jump-auth-refused', 'jump-unreachable'], ['target-firewall-dropping', 'target-unreachable']] six tests 0 the tunnel entry point's bind address (24 other hosts in the subnet, 3 tunnels open) loopback other hosts that can reach the tunnel entry 0 | indirect (host, target service) pairs 0 all interfaces other hosts that can reach the tunnel entry 24 | indirect (host, target service) pairs 72 candidate set sweep 9 candidates: target-auth-refused removed end-to-end eliminated 7 remaining 2 | indistinguishable group 0 largest 0 10 candidates: base list end-to-end eliminated 8 remaining 2 | indistinguishable group 0 largest 0 11 candidates: jump-session-restricted added end-to-end eliminated 9 remaining 2 | indistinguishable group 1 largest 2 ``` #### Three Numbers **Oracle:** the real cause is `target-firewall-dropping` — the target's filter silently drops the packet. **Test:** the end-to-end attempt says "channel timed out"; the attempt made from the jump host to the target says "timeout"; the local listener list says the tunnel is bound to the expected address. **Candidates eliminated:** end-to-end **8**, hop-to-target **9**, local listener **2**, name location **1**. The end-to-end attempt is the strongest single test in this setup: it produces seven distinct answers and, in the worst case, leaves only two candidates. Even so, it cannot find the oracle on its own, because not being able to reach the target and the target's filter dropping the packet look **the same** from inside the tunnel. What separates them is the attempt made from the jump host: there, "unreachable" and "timeout" are different answers, and that test alone eliminates nine candidates. The placement rule that follows is this: a jump host is as much an **observation point** as it is a gate. It sits at the point nearest the target, and an attempt made there removes every leg in between from the equation. Being able to open a session on a jump host means a nine-candidate elimination in diagnosis. #### Order and Indistinguishable Groups The second table compares two fixed orders. The layered order proceeds from client to target and spends a total of **34** steps for the ten faults. The order that puts the end-to-end attempt first finishes in **28** steps. Both can be applied without knowing the oracle; the six-step difference between them comes from the end-to-end attempt producing seven distinct answers. This result does not contradict what was measured on a single machine. There, the connection attempt gave four distinct answers; here, the attempt made through the tunnel rises to seven, because the tunnel carries back the failure at each stop along the path as a **separate message**. A test's power is not in the tool's name, it is in how many distinct answers it produces. The third table shows where the elimination power runs out. Looking only at end-to-end, three groups cannot be separated: two local faults, two first-leg faults, and two silent target faults. Adding the jump session resolves the second group, adding the hop-to-target attempt the third. The three local tests, on the other hand, leave **two** groups; none of them can get past the client. When all six tests are used together, the indistinguishable group count is **0**. #### Candidate Set Sweep The bottom table tries the candidate list at three sizes. When `target-auth-refused` is removed, the number of candidates the end-to-end attempt eliminates drops from eight to seven, and what remains is still two. When `jump-session-restricted` is added to the list, the eliminated count **rises** to nine and what remains is still two; against this, a group that cannot be separated even with all six tests **is born**, because the new candidate gives the same answer as `forwarding-disabled` on every test. One detail falls outside this measurement and needs to be stated. The tunnel's channel error appears **once, on the client's terminal**, and is written nowhere. If the tunnel was opened in the background, that single line is not seen either; what remains is only a connection that does not work. The record of the session at the hop is in the jump host's log, and whether the channel opening and closing is recorded there depends on the verbosity level set there. **The evidence does not wait:** looking for the channel error afterward gives a result only if the verbosity level was sufficient at the moment of the fault. These three lines show the previous lesson's rule from another direction. There, the elimination count stayed fixed while what remained grew; here, as the elimination count grows, **distinguishability** drops. In both cases, elimination on its own is not a measure of success. A candidate set is only considered resolved if the tests' answer patterns can separate it one by one, and the measure of this is the count of indistinguishable groups. #### Summary - A jump host splits the path in two: in tunnel form a mouth opens on the client, in hop form the session opens directly on the target and the jump host only carries. - The target's name resolves on the jump host; if the same name is written in the client's configuration, the client tries to resolve it and a name resolution fault that does not exist on the server appears. - The tunnel entry point's bind address determines the exposed surface: at loopback, indirectly reachable pairs are **0**; at all interfaces, for 24 hosts and 3 tunnels in the subnet, **72**. - Across ten candidate causes, the end-to-end attempt gives seven distinct answers and eliminates **8** candidates; the attempt made from the jump host to the target eliminates **9** and resolves it on its own; the location of name resolution eliminates only **1**. - The layered order spends **34** steps, the order that puts the end-to-end attempt first spends **28**; both can be applied without knowing the oracle. - When one item is added to the candidate list, the eliminated count rises from eight to nine while the indistinguishable group count rises from zero to one; the measure is not elimination but distinguishability. #### Next Step This lesson's oracle was the target's filter, and looked at from inside the tunnel, that filter could not be told apart from not reaching the target at all. A silently dropped packet is the hardest event to trace in this course, because it sends nothing back. The next lesson goes inside that filter: it builds which stops a packet passes through inside a machine, in what order rules are evaluated, and which reading can show at which stop a packet was dropped. ### The Packet-Filtering Framework Source: https://academia.sh/en/courses/linux-network-administration/packet-filtering-framework The seven hops a packet passes through inside the machine, the table-and-chain distinction, and rule evaluation order are established; a thirty-four-line rule list eliminates zero, a counter reading eliminates zero on a loaded machine and six on a quiet one, and open tracing eliminates eight. The previous lesson's oracle was the target's filter, and looked at from inside the tunnel, that filter could not be told apart from not reaching the target at all. A silently dropped packet sends nothing back; there is a timeout for the sender, no event at all for the destination. This lesson goes inside that filter. The question to ask is a question of place: how far into this machine did a packet get, and **at which stop** was the decision made. The answer to this question is not written in a rule list. The list shows what **could** happen; it does not show which rule actually ran, and the gap between these two sentences is what this lesson measures. #### Hops, Tables, and Chains The packet-filtering framework is made of **hooks** placed at fixed points in the kernel's network stack. Each hook holds one or more **chains**, and chains are grouped by **tables**. A table is a purpose: raw processing, marking, address translation, filtering. A chain is that purpose's rule list at a particular hook. When more than one table has a chain at the same hook, their order is fixed by table priority. The packet's path splits in two after the routing decision. The dump below is not executed; it is written to show the order among the hooks: ```text # example dump, not executed # a packet headed to a local process interface -> prerouting-raw -> prerouting-nat -> routing-decision -> input-mangle -> input-filter -> service # a packet forwarded to another interface interface -> prerouting-raw -> prerouting-nat -> routing-decision -> forward-filter -> postrouting -> output-interface ``` The two paths are identical up to the routing decision and never merge again after it. This is the source of a common mistake in diagnosis: the forward chain is examined for a packet that never reaches the local service, when the packet never passed through there at all. Or the reverse happens, and a rule meant for a packet to be forwarded is written into the input chain. **The chain's name states where the decision is made**; where the decision is made depends on the path the packet follows. A chain has two components: ordered rules and a **policy**. Rules are evaluated top to bottom, the **first matching rule** makes the decision, and the rest of the chain does not run. If no rule matches, the policy is applied. The dump below is also not executed and is not a real framework's syntax; it is a simplified notation meant to show how the chain, policy, and order concepts are laid out: ```text # example dump, not executed, not real syntax table filter chain input policy drop 1 state established or related -> accept 2 interface loopback -> accept 3 destination port -> accept chain forward policy drop chain output policy accept ``` What rule evaluation order changes **within a single list** was measured in the access control lists lesson of the **Cybersecurity** curriculum. What is measured here is not order within a list but **order among hops**: once a packet finds its decision at one hop, the chains of the following hops never run at all, and a rule written there does nothing. #### At Which Hop Is the Decision Made The setup is this: a remote client cannot reach a service on this machine. There are nine candidate causes, and each makes the decision **at a specific hop**. - **EF21** — The candidate-cause list has **nine** items and is exhaustive. Three are in the input-filter chain, two on the forwarding path, one at the raw hook, one at the marking hook, one before the interface, one at the service. - **EF22** — The oracle is chosen as `input-default-refused`: no rule matches, and the chain policy refuses the packet. - **EF23** — The rule set is **the same in all nine cases** and is **34** lines; what changes is the packet's fate, not the configuration. - **EF24** — Eight readings are defined, and four of them are measured under two conditions each: counter reading on a loaded and a quiet machine, tracing on and off. - **EF25** — On a loaded machine, background traffic increases every chain's counters; the counter difference cannot be tied to the test packet. - **EF26** — Tracing must have been turned on **before** the fault; while off, the lines it produces are **0**. - **EF27** — Tracing reports the hop and the reason together; two candidates that drop at the same hop for the same reason still cannot be separated by it. - **EF28** — An accept rule is added to the input-filter chain in two positions: at the head and at the tail. A rule added at the tail is never evaluated if a rule before it already matches. - **EF29** — The candidate set sweep is done with eight-, nine-, and ten-candidate lists. - **EF30** — All counts are exhaustive; there is no randomness or seed. ```python """The packet's stops inside the machine: at which hop the decision was made, and which reading sees it.""" LOCAL = ("interface", "prerouting-raw", "prerouting-nat", "routing-decision", "input-mangle", "input-filter", "service") FORWARD = ("interface", "prerouting-raw", "prerouting-nat", "routing-decision", "forward-filter", "postrouting", "output-interface") PATH = {"local": LOCAL, "forward": FORWARD} CHAIN = {"prerouting-raw", "prerouting-nat", "input-mangle", "input-filter", "forward-filter", "postrouting"} CANDIDATE = { "packet-never-arrived": ("local", 0, "none"), "prerouting-raw-dropped": ("local", 1, "rule"), "input-mangle-dropped": ("local", 4, "rule"), "input-filter-rule-dropped": ("local", 5, "rule"), "input-default-refused": ("local", 5, "policy"), "state-rule-missing": ("local", 5, "rule"), "nat-wrong-target": ("forward", 4, "rule"), "routing-wrong": ("forward", 4, "rule"), "service-not-listening": ("local", 6, "service"), } RULE_LINES = 34 ORACLE = "input-default-refused" def last_hop(a): path, i, _ = CANDIDATE[a] return PATH[path][i] def unreached(a): path, i, _ = CANDIDATE[a] return len(PATH[path]) - (i + 1) TEST = { "rule-list": lambda a: f"{RULE_LINES}-lines", "interface-capture": lambda a: "did-not-arrive" if CANDIDATE[a][1] == 0 else "arrived", "egress-capture": lambda a: "exited" if CANDIDATE[a][0] == "forward" else "did-not-exit", "connection-attempt": lambda a: ("refused" if CANDIDATE[a][2] == "service" else "timeout"), "counter-loaded": lambda a: "all-counters-increased", "counter-quiet": lambda a: (last_hop(a) if last_hop(a) in CHAIN else "no-counter"), "trace-off": lambda a: "no-line", "trace-on": lambda a: f"{last_hop(a)}/{CANDIDATE[a][2]}", } LINES = {"rule-list": RULE_LINES, "interface-capture": 6, "egress-capture": 6, "connection-attempt": 1, "counter-loaded": 34, "counter-quiet": 34, "trace-off": 0, "trace-on": 12} def eliminate(candidates, s, answer): return tuple(a for a in candidates if TEST[s](a) == answer) def group(tests, candidates): o = {} for a in candidates: o.setdefault(tuple(TEST[s](a) for s in tests), []).append(a) return [sorted(v) for v in o.values() if len(v) > 1] A = tuple(CANDIDATE) print("candidate causes:", len(A), "| tests:", len(TEST), "| oracle:", ORACLE) print() print("candidate path last hop unreached hops") for a in A: print(f" {a:26s} {CANDIDATE[a][0]:7s} {last_hop(a):18s} {unreached(a):9d}") print(" total unreached hops:", sum(unreached(a) for a in A)) print() print("test lines distinct answers eliminated remaining") for s in TEST: remaining = len(eliminate(A, s, TEST[s](ORACLE))) print(f" {s:18s} {LINES[s]:5d} {len({TEST[s](a) for a in A}):11d}" f" {len(A) - remaining:7d} {remaining:6d}") print() print("tool set lines indistinguishable group") for ad, k in (("rule list alone", ["rule-list"]), ("list + connection attempt", ["rule-list", "connection-attempt"]), ("four readings on a loaded machine", ["rule-list", "interface-capture", "connection-attempt", "counter-loaded"]), ("four readings on a quiet machine", ["rule-list", "interface-capture", "connection-attempt", "counter-quiet"]), ("trace on", ["trace-on"])): o = group(k, A) print(f" {ad:34s} {sum(LINES[s] for s in k):5d} {len(o):8d}") if o: for x in o: print(f" {x}") print() print("how many faults an accept rule added to the input-filter chain fixes") for ad, cond in ( ("at the head of the chain", lambda a: CANDIDATE[a][0] == "local" and CANDIDATE[a][1] == 5), ("at the tail of the chain", lambda a: CANDIDATE[a][0] == "local" and CANDIDATE[a][1] == 5 and CANDIDATE[a][2] == "policy")): d = [a for a in A if cond(a)] print(f" {ad:16s} fixed {len(d)} / {len(A)}", d) print() print("candidate set sweep") CANDIDATE["input-filter-wrong-interface"] = ("local", 5, "rule") for ad, k in (("8 candidates: state-rule-missing removed", tuple(a for a in A if a != "state-rule-missing")), ("9 candidates: base list", A), ("10 candidates: input-filter-wrong-interface added", A + ("input-filter-wrong-interface",))): remaining = len(eliminate(k, "counter-quiet", TEST["counter-quiet"](ORACLE))) o = group(["trace-on"], k) print(f" {ad:42s} quiet counter eliminated {len(k) - remaining:2d} remaining {remaining:2d}" f" | group trace cannot separate {len(o)}" f" largest {max((len(x) for x in o), default=0)}") ``` ``` candidate causes: 9 | tests: 8 | oracle: input-default-refused candidate path last hop unreached hops packet-never-arrived local interface 6 prerouting-raw-dropped local prerouting-raw 5 input-mangle-dropped local input-mangle 2 input-filter-rule-dropped local input-filter 1 input-default-refused local input-filter 1 state-rule-missing local input-filter 1 nat-wrong-target forward forward-filter 2 routing-wrong forward forward-filter 2 service-not-listening local service 0 total unreached hops: 20 test lines distinct answers eliminated remaining rule-list 34 1 0 9 interface-capture 6 2 1 8 egress-capture 6 2 2 7 connection-attempt 1 2 1 8 counter-loaded 34 1 0 9 counter-quiet 34 5 6 3 trace-off 0 1 0 9 trace-on 12 7 8 1 tool set lines indistinguishable group rule list alone 34 1 ['input-default-refused', 'input-filter-rule-dropped', 'input-mangle-dropped', 'nat-wrong-target', 'packet-never-arrived', 'prerouting-raw-dropped', 'routing-wrong', 'service-not-listening', 'state-rule-missing'] list + connection attempt 35 1 ['input-default-refused', 'input-filter-rule-dropped', 'input-mangle-dropped', 'nat-wrong-target', 'packet-never-arrived', 'prerouting-raw-dropped', 'routing-wrong', 'state-rule-missing'] four readings on a loaded machine 75 1 ['input-default-refused', 'input-filter-rule-dropped', 'input-mangle-dropped', 'nat-wrong-target', 'prerouting-raw-dropped', 'routing-wrong', 'state-rule-missing'] four readings on a quiet machine 75 2 ['input-default-refused', 'input-filter-rule-dropped', 'state-rule-missing'] ['nat-wrong-target', 'routing-wrong'] trace on 12 2 ['input-filter-rule-dropped', 'state-rule-missing'] ['nat-wrong-target', 'routing-wrong'] how many faults an accept rule added to the input-filter chain fixes at the head of the chain fixed 3 / 9 ['input-filter-rule-dropped', 'input-default-refused', 'state-rule-missing'] at the tail of the chain fixed 1 / 9 ['input-default-refused'] candidate set sweep 8 candidates: state-rule-missing removed quiet counter eliminated 6 remaining 2 | group trace cannot separate 1 largest 2 9 candidates: base list quiet counter eliminated 6 remaining 3 | group trace cannot separate 2 largest 2 10 candidates: input-filter-wrong-interface added quiet counter eliminated 6 remaining 4 | group trace cannot separate 2 largest 3 ``` #### Three Numbers **Oracle:** the decision is made at the `input-filter` hop; because no rule matches, it is made by the **policy**. **Test:** the rule list prints thirty-four lines; the counter reading on a loaded machine gives thirty-four numbers; the same reading on a quiet machine gives the chain name; open tracing says "input-filter/policy" in twelve lines. **Candidates eliminated:** rule list **0**, loaded counter **0**, tracing off **0**, connection attempt **1**, quiet counter **6**, tracing on **8**. The three zeros above are this lesson's result. The rule list is a thirty-four-line output, and it eliminates none of the nine candidates, because it is **the same list in all nine cases**. The list shows the configuration, not the packet's fate. The output with the highest count sits in the same row as the one with the lowest elimination power: **more output does not mean better diagnosis.** The connection attempt prints one line and eliminates one candidate. Its ratio is infinitely better than the rule list's, but its absolute value is low, and the reason was established in the shared reference: a dropped packet produces a timeout, a refused packet gives a separate answer. Eight of nine candidates produce a timeout. #### What the Counter Proves The difference between the counter reading's two lines is this lesson's most practical finding. On a quiet machine, the counter gives five distinct answers and eliminates **six** candidates: whichever chain's counter increased is where the decision was made. On a loaded machine, the same reading gives a **single** answer and eliminates **zero** candidates, because background traffic increases every chain's counters. The counter difference exists; it just **cannot be shown** to belong to the test packet. For this reason, a counter reading cannot be written as a standalone procedure on its own. To become measurable, either traffic must stop, or the counter must be reset before the test and only one packet sent; the second cannot always be done on a running system. What the counter tells you depends on **how quiet** the machine is, and this is a property of the environment, not of the tool. Tracing's two lines give the same lesson on the time axis. On, it produces twelve lines and eliminates eight candidates; off, **zero lines** and zero candidates. The difference is a setting, and that setting must have been turned on **before** the fault. Turning tracing on after the fault does not bring back the packet from the past. **The evidence does not wait.** Even open tracing cannot separate everything. Two groups remain: a rule fault dropping for the same reason at the same chain versus the state rule being missing, and the two faults dropping on the forwarding path. Tracing states the hop and the reason; stating **which rule** matched requires a separate level of detail. #### Where Adding a Rule Actually Helps The bottom table counts the framework's most expensive misunderstanding. When an accept rule is added to the input-filter chain, how many of nine faults are fixed: **3** if the rule is added to the **head** of the chain, **1** if added to the **tail**. The gap between the two numbers comes from the first-matching-rule rule. An accept rule added at the tail is never evaluated at all if a drop rule matching before it exists; it fixes only the case where no rule matches at all — that is, the case where the policy kicks in. The rule was added, the list got longer, the output came back "successful," and the fault continues. For the remaining six faults, position makes no difference at all, because the packet **never reaches** that chain in the first place. The top table's last column counts this: for nine faults, the total of unreached hops is **20**. If the packet was dropped at the raw hook, the next five hops never ran; if dropped at the marking hook, the next two never ran. Every rule written at those hops is a **dead rule**, however correctly it is written. The order rule that follows is this: before adding a rule, find out **at which hop** the decision is being made. When this order is reversed — adding the rule first — what results is not an error message but a fix that does not work and a growing rule set. #### Candidate Set Sweep The bottom table tries the candidate list at three sizes, and the result looks in the same direction as the previous two lessons. The number of candidates the quiet counter eliminates is **6** in all three lists; what remains is **2**, **3**, and **4**. As the candidate list grows, the same reading eliminates the same number of candidates and leaves more behind. The largest group tracing cannot separate rises from two members to three, because the added candidate drops at the same hop for the same reason. #### Summary - The packet-filtering framework is made of fixed hooks; each hook holds chains grouped by table, and after the routing decision the local path and the forward path never merge again. - In a chain, the first matching rule makes the decision and the rest does not run; if no rule matches, the policy applies. - A thirty-four-line rule list eliminates **none** of nine candidates, because it is the same in all nine cases. On a loaded machine the counter reading also eliminates **0**; on a quiet machine **6**, with tracing on **8**. - Tracing produces **0 lines** while off, and turning it on after the fault does not bring back the packet from the past. - An accept rule added to the input-filter chain fixes **3** faults at the head, **1** at the tail; for the remaining six, the packet never reaches that chain at all. The total of unreached hops across nine faults is **20**. #### Next Step This lesson asked where the decision was made and chose the policy as the oracle: no rule matched, so the chain's default kicked in. The next lesson takes up that default itself. A default-deny policy is a security decision, and the **order** it is applied in is an operational one; applied in the wrong order, it can cut off the administrator's own access. What will be measured is how many of the application orders produce this result, and which paths bring the cost down to zero. ### Firewall Rules Source: https://academia.sh/en/courses/linux-network-administration/firewall-rules The default-deny policy's five steps are applied in a hundred twenty orders; forty produce a full lockout, twenty a partial one, and because the final rule set is the same across all hundred twenty orders, a dry run prevents zero of them; on writing errors a dry run eliminates five candidates and an end-to-end attempt eliminates five. The previous lesson asked at which hop the decision was made and chose the policy as the oracle: because no rule matched, the chain's default kicked in. That default is not an accident, it is a choice. This lesson takes up that choice and the work of setting it up. A default-deny policy is a single line, and **when** that line is written produces more consequences than what it writes. What this lesson measures is not the content of the rules but the **order they are applied in**; and choosing that order wrong can cut the one path between the administrator and the machine. #### Default Deny and Legitimate Flows **Default deny** means setting a chain's policy to "refuse everything that does not match." Its opposite is default allow: everything that does not match passes, and you have to write down, one by one, every case you want to forbid. The difference between the two is not a difference in rule count, it is a difference in **error direction**. Under default allow, a forgotten rule produces an opening; under default deny, a forgotten rule produces an outage. An outage is seen; an opening is not. There are legitimate flows that must be kept alive while default deny is being set up, and in this setup there are four. The **loopback** flow is the machine's own processes talking to one another. The **established connection** flow is the return packets of sessions already open. The **management** flow is the administrator's way of reaching this machine. The **service** flow is what the machine offers outward. Each is kept alive by one rule. There are five steps, four of them allow rules and one the policy. The dump below is not executed and is not a runnable command sequence; it is written only to show the names of the steps and the difference between two orderings: ```text # example dump, not a runnable sequence # safe order: the deny policy is left for LAST 1 loopback-allow 2 established-connection-allow 3 management-allow 4 service-allow 5 default-deny # locking order: deny before management and established-connection allows 1 default-deny <- the remote session drops at this step 2 ... <- the remaining steps cannot be applied ``` **Lockout** is the administrator cutting off their own access. This word is the same as the `deadlock` counterpart in the **Operating System Concepts** course, but the event is not the same: there, two threads wait on each other and neither can proceed; here, there is a single administrator who has closed their own channel. What they share is only that the result looks irreversible. #### A Hundred Twenty Orders The setup is this: five steps are going to be applied on a machine, and the administrator is connected remotely. The order of the steps is free; every order is counted exhaustively. - **EF31** — The five steps' **120** orders are produced by exhaustive count; there is no sampling and no seed. - **EF32** — The administrator's session stays alive when the deny policy is applied if at least one of the **management allow** or **established-connection allow** rules is already in place. - **EF33** — Once the session drops, the remaining steps **cannot be applied**; the order is cut off there. - **EF34** — If deny is applied with the established-connection allow granted but management allow not granted, the current session survives, but **no new connection can be established**; this case is called a partial lockout. - **EF35** — The rollback timer is set to **120 seconds**; when the time elapses, the previous rule set is restored. - **EF36** — There are **seven candidates** in the rule-writing measurement: six writing errors and the error-free case. - **EF37** — A dry run produces the rule set that would be applied, without applying it, and sees **structural** errors; because it does not know intent, it cannot see a misspelled port. - **EF38** — An end-to-end attempt probes all four legitimate flows; for this reason it sees all six errors, but it requires the rule set to have already been **applied**. - **EF39** — The candidate set sweep is done with six-, seven-, and eight-candidate lists. - **EF40** — All counts are exhaustive; there is no randomness or seed. ```python """Default deny: application order and rule-writing errors. Part 1 - all 120 orders of the five steps are counted exhaustively; how many orders produce a lockout. Part 2 - seven candidates (six writing errors and the error-free case), four readings. """ from itertools import permutations STEP = ("loopback-allow", "established-connection-allow", "management-allow", "service-allow", "default-deny") ROLLBACK = 120 def apply(order): """If neither management nor established-connection allow is in place when the deny policy is applied, the remote session drops; the remaining steps cannot be applied.""" present = set() for i, a in enumerate(order): present.add(a) if a == "default-deny": if not ({"management-allow", "established-connection-allow"} & present): return "full-lockout", i + 1, len(order) - (i + 1) if "management-allow" not in present: return "partial-lockout", i + 1, 0 return "safe", len(order), 0 ORDERS = list(permutations(STEP)) RESULT = [apply(s) for s in ORDERS] COUNT = {} for kind, _, _ in RESULT: COUNT[kind] = COUNT.get(kind, 0) + 1 print("all orders of five steps:", len(ORDERS)) for kind in ("safe", "partial-lockout", "full-lockout"): print(f" {kind:18s} {COUNT.get(kind, 0):3d} / {len(ORDERS)}" f" ratio {COUNT.get(kind, 0) / len(ORDERS):.4f}") print(" total unapplied steps under full lockout:", sum(u for t, _, u in RESULT if t == "full-lockout")) print(" in how many distinct forms does the final rule set end up:", len({frozenset(s) for s in ORDERS})) print() LOCKOUT_TOTAL = COUNT["full-lockout"] print("application method lockouts external intervention downtime (s)" " prevented by dry run") for ad, lockouts, intervention, downtime, dry_run in ( ("plain application", LOCKOUT_TOTAL, LOCKOUT_TOTAL, 0, 0), ("application after dry run", LOCKOUT_TOTAL, LOCKOUT_TOTAL, 0, 0), ("timer-based rollback", LOCKOUT_TOTAL, 0, LOCKOUT_TOTAL * ROLLBACK, 0), ("separate management channel", LOCKOUT_TOTAL, 0, 0, 0), ("atomic load", 0, 0, 0, 0)): print(f" {ad:23s} {lockouts:10d} {intervention:16d} {downtime:13d} {dry_run:27d}") print() FLAW = { "syntax-error": (True, True, "management-flow-cut"), "rule-in-wrong-chain": (False, True, "management-flow-cut"), "rule-order-shadows": (False, True, "service-flow-cut"), "state-rule-missing": (False, True, "return-path-cut"), "wrong-interface-name": (False, True, "management-flow-cut"), "wrong-port": (False, False, "service-flow-cut"), "no-error": (False, False, "everything-passed"), } ORACLE = "wrong-port" TEST = { "syntax-check": lambda a: "error" if FLAW[a][0] else "clean", "rule-list": lambda a: ("no-rule" if a == "syntax-error" else "different-chain" if a == "rule-in-wrong-chain" else "in-place"), "dry-run": lambda a: a if FLAW[a][1] else "clean", "end-to-end-test": lambda a: FLAW[a][2], } RISK = {"syntax-check": 0, "rule-list": 0, "dry-run": 0, "end-to-end-test": sum(1 for a in FLAW if FLAW[a][2] == "management-flow-cut")} A = tuple(FLAW) def eliminate(candidates, s, answer): return tuple(a for a in candidates if TEST[s](a) == answer) def group(tests, candidates): o = {} for a in candidates: o.setdefault(tuple(TEST[s](a) for s in tests), []).append(a) return [sorted(v) for v in o.values() if len(v) > 1] print("candidates:", len(A), "| oracle:", ORACLE) print() print("reading distinct answers eliminated remaining risk of cutting management") for s in TEST: remaining = len(eliminate(A, s, TEST[s](ORACLE))) print(f" {s:18s} {len({TEST[s](a) for a in A}):11d} {len(A) - remaining:7d}" f" {remaining:6d} {RISK[s]:20d}") print() print("reading set indistinguishable group") for ad, k in (("syntax check alone", ["syntax-check"]), ("syntax + rule list", ["syntax-check", "rule-list"]), ("dry run", ["dry-run"]), ("end-to-end test", ["end-to-end-test"]), ("dry run + end-to-end test", ["dry-run", "end-to-end-test"])): o = group(k, A) print(f" {ad:34s} {len(o):8d}", o if o else "") print() print("candidate set sweep") FLAW["wrong-source-range"] = (False, False, "service-flow-cut") for ad, k in (("6 candidates: state-rule-missing removed", tuple(a for a in A if a != "state-rule-missing")), ("7 candidates: base list", A), ("8 candidates: wrong-source-range added", A + ("wrong-source-range",))): remaining = len(eliminate(k, "dry-run", TEST["dry-run"](ORACLE))) o = group(["dry-run", "end-to-end-test"], k) print(f" {ad:38s} dry run eliminated {len(k) - remaining:2d} remaining {remaining:2d}" f" | group the two readings together cannot separate {len(o)}" f" largest {max((len(x) for x in o), default=0)}") ``` ``` all orders of five steps: 120 safe 60 / 120 ratio 0.5000 partial-lockout 20 / 120 ratio 0.1667 full-lockout 40 / 120 ratio 0.3333 total unapplied steps under full lockout: 140 in how many distinct forms does the final rule set end up: 1 application method lockouts external intervention downtime (s) prevented by dry run plain application 40 40 0 0 application after dry run 40 40 0 0 timer-based rollback 40 0 4800 0 separate management channel 40 0 0 0 atomic load 0 0 0 0 candidates: 7 | oracle: wrong-port reading distinct answers eliminated remaining risk of cutting management syntax-check 2 1 6 0 rule-list 3 2 5 0 dry-run 6 5 2 0 end-to-end-test 4 5 2 3 reading set indistinguishable group syntax check alone 1 [['no-error', 'rule-in-wrong-chain', 'rule-order-shadows', 'state-rule-missing', 'wrong-interface-name', 'wrong-port']] syntax + rule list 1 [['no-error', 'rule-order-shadows', 'state-rule-missing', 'wrong-interface-name', 'wrong-port']] dry run 1 [['no-error', 'wrong-port']] end-to-end test 2 [['rule-in-wrong-chain', 'syntax-error', 'wrong-interface-name'], ['rule-order-shadows', 'wrong-port']] dry run + end-to-end test 0 candidate set sweep 6 candidates: state-rule-missing removed dry run eliminated 4 remaining 2 | group the two readings together cannot separate 0 largest 0 7 candidates: base list dry run eliminated 5 remaining 2 | group the two readings together cannot separate 0 largest 0 8 candidates: wrong-source-range added dry run eliminated 5 remaining 3 | group the two readings together cannot separate 1 largest 2 ``` #### The Cost of Order The top table splits the hundred twenty orders into three groups. **60** orders are safe, **20** produce a partial lockout, **40** produce a full lockout. A third. When no preference is made between the deny policy, management allow, and established-connection allow, which of these three steps comes first is equally likely, and deny coming first is a lockout. Partial lockout's twenty orders are the most treacherous group. Established-connection allow has been granted, deny has been applied, management allow has not yet been written. The administrator's session is **alive**; commands run, output comes back. There is no symptom at all. When that session closes for any reason — and the reason can be a network interruption, closing the terminal, or a timeout — there is no way back. **A session staying alive is not proof that access continues.** Full lockout's forty orders carry one more cost: the total of unapplied steps is **140**. When the order is cut off, the rule set is left half-finished, and a half-finished set usually shuts out both the administrator and the service at once. The fault is not a single access fault; it is a service outage at the same time. The top table's last row is this lesson's most important finding: **the final rule set is the same across all hundred twenty orders** — a single form. The set of steps to be sequenced does not change, only their order does. The direct consequence of this is that a dry run, which produces and examines the final rule set without applying it, sees **none** of these lockouts. This is why the second table's last column is zero from top to bottom. **The danger is not in the final state, it is in the intermediate states.** #### Paths That Prevent Lockout The second table places four paths side by side by their cost. **Plain application** means forty external interventions for forty lockouts; each intervention means console-level access or physically going to the machine. **Application after a dry run** gives the same numbers; a dry run is worthless for this class of fault. **Timer-based rollback** does not prevent the forty lockouts, but it brings external intervention down to **zero**: when the hundred-twenty-second timer elapses, the previous rule set is restored and the administrator gets back in. Its cost is a total of **4800** seconds of downtime. A **separate management channel** also does not prevent the lockout, brings intervention down to zero, and produces no downtime; its cost is that channel having to exist at all times. **Atomic load** changes the rule set in a single operation; because no intermediate state is created, the lockout count becomes **0**. The ranking of the four paths follows from this table. The path that produces no intermediate state is best, because it removes the problem at its source. The path that keeps the intermediate state it produces short comes second. The path that makes the intermediate state tolerable comes third. The path that only examines the intermediate state does nothing at all for this class. A lockout also carries no evidence of its own, and this is this lesson's form of the course's third claim. The moment the session drops, all the administrator has left is a frozen screen on the terminal; the line that would say what happened on the machine is inside the machine, and the machine cannot be entered. The trace of a dropped management packet is also **zero lines**, exactly as counted in the previous lesson, unless recording was already on. This is why the timer's duration is chosen not for reading a log and deciding, but for **coming back**. #### Three Numbers The bottom measurement looks at rule writing: the set is applied in the correct order, but the written rule itself can have an error. **Oracle:** of seven candidates, the real one is `wrong-port` — the rule is written, its syntax is correct, its chain is correct, its order is correct, and the port inside it is wrong. **Test:** the syntax check says "clean," the rule list shows the rule as "in place," the dry run says "clean," the end-to-end attempt says "service flow cut." **Candidates eliminated:** syntax check **1**, rule list **2**, dry run **5**, end-to-end attempt **5**. The last two readings eliminate the same number of candidates, and the two-candidate residue they leave behind is **different**. The pair the dry run leaves is `no-error` and `wrong-port`; that is, when the dry run says "clean," it cannot distinguish between being genuinely clean and carrying an error it cannot see. The pair the end-to-end attempt leaves is `rule-order-shadows` and `wrong-port`; both cut the same flow. When the two readings are used together, the indistinguishable group count is **0**. The column on the right says why both are needed. The end-to-end attempt's risk of cutting the management flow is **3**: in three of seven candidates, the set that has to be applied for the attempt to be possible at all closes the administrator's own path. The dry run's risk is **0**. The correct order follows from this: the risk-free reading first, then the risky attempt under the protection of a timer or a separate channel. A dry run does not replace the attempt; **it makes the attempt cheaper**, because it eliminates, before any attempt is made, the five errors it is able to catch. #### Candidate Set Sweep The bottom table again tries three sizes. When `state-rule-missing` is removed, the number of candidates the dry run eliminates drops from five to four, and what remains is still two. When `wrong-source-range` is added to the list, the eliminated count **stays at 5**, what remains rises to three, and even when the two readings are used together, an unseparated group **is born**: the new candidate also escapes the dry run and cuts the same flow. The pattern stayed the same throughout the topic. As the candidate list grows, the elimination count stays fixed, and the remaining ambiguity grows. Every time an error class the dry run cannot see is added to the list, the meaning of a "clean" answer weakens by one more candidate. #### Summary - The default-deny policy needs four allow rules keeping four legitimate flows alive: loopback, established connection, management, and service. - Of the five steps' **120** orders, **60** are safe, **20** produce a partial lockout, **40** a full lockout; the total of unapplied steps under a full lockout is **140**. - Under a partial lockout, the current session survives and gives no symptom at all; a session staying alive is not proof that access continues. - The final rule set is the same across all hundred twenty orders; for this reason a dry run prevents **0** of the order-caused lockouts. Timer-based rollback and a separate management channel zero out external intervention, atomic load zeros out the lockout itself. - On rule-writing errors, a dry run eliminates **5** candidates and an end-to-end attempt eliminates **5**, and what they leave behind differs; together they bring the indistinguishable group down to **0**. The end-to-end attempt's risk of cutting management is **3**, the dry run's is **0**. #### Next Step This lesson counted how a machine closes its own gate. The next lesson looks in the opposite direction: when the same machine sits at the exit of a network, it does address translation for the machines behind it and forwards incoming connections inward. Every forwarding rule opens a window next to the door that was closed. The course's final lesson measures the number of those windows, the damage they do to diagnosis, and how long a translation record stays alive. ### Address Translation Source: https://academia.sh/en/courses/linux-network-administration/address-translation The surface that port forwarding opens outward is counted point by point: four rules open thirteen points and reach eight pairs today, the translated log line eliminates zero of twelve candidate sources, and at the six-hundredth second only eleven of forty translation entries remain resolvable. The previous lesson counted how a machine could lock its own door: forty of the hundred and twenty orderings of five steps left the administrator outside. This lesson looks the other way. When the same machine stands at a network's exit, it does not only hold the door; it speaks on behalf of the machines behind it and carries some inbound connections in. Every forwarding rule opens a window next to the door it closes. The course's last lesson counts those windows and measures the damage they do to diagnosis. This lesson does not give an instruction in the form "open this"; it writes what opening means in **how many pairs**. #### The Gateway's Two Jobs **Address translation** (NAT) is the changing, at the gateway, of the address or port fields in a packet's header. It has two directions. **Source translation** applies to packets going out from inside: the internal host's address is replaced with the gateway's address, and the gateway keeps a translation entry so it can send the returning packet back to the correct internal host. **Destination translation** applies to packets coming in from outside: a packet arriving at a port on the gateway is forwarded to a point on an internal host. The second is called **port forwarding**. The types of translation, the translation table's five-component key, its capacity limit, and the breaking of end-to-end connectivity were established in the **Computer Networks** course and are not repeated here. This lesson asks two things: how large is the surface this rule opens, and what information does this rule erase. Placement is also this course's subject. Source translation happens at the stop **after** the routing decision; destination translation, at the stop **before** it. Because destination translation runs first, filtering rules see the packet in its **translated** form; an operator who writes a rule against the external address matches nothing, and there is no error message to show for it. #### Surface, Identity, and the Record The setup is this: an internal network sits behind a gateway, and four forwarding rules are written. - **EF41** — The internal network has **12** hosts and **5** services per host; the internal (host, service) pair count is **60**. - **EF42** — With no translation, the internal pairs directly reachable from outside are **0**. - **EF43** — Four rules are written: three open a single point each, one opens a **10**-point range. - **EF44** — On the host targeted by the range rule, **5** services are listening today; the remaining five points are closed today, and the rule covers them too. - **EF45** — The oracle is the true source host, and `internal-host-07` is chosen. - **EF46** — The target server's log writes a single source field; with translation in place, this field is the gateway's address. - **EF47** — A translation entry drops **120 seconds** after the connection closes. - **EF48** — The window is **600** seconds and there are **40** connections; connection `i` starts at multiples of the fifteenth second, and its duration is `30 + (i mod 7) * 10` seconds. - **EF49** — The candidate set sweep is done with internal host counts of 8, 12, and 16. - **EF50** — All numbers are exact counts; there is no randomness and no seed. ```python """Address translation: the surface opened, the loss of source identity, the record's lifetime.""" INTERNAL_HOSTS, SERVICES = 12, 5 TIMEOUT, WINDOW, CONNECTIONS = 120, 600, 40 RULE = (("single-a", 1, 1), ("single-b", 1, 1), ("single-c", 1, 1), ("range-d", 10, 5)) print("internal network:", INTERNAL_HOSTS, "hosts x", SERVICES, "services =", INTERNAL_HOSTS * SERVICES, "internal (host, service) pairs") print("pairs directly reachable from outside (no translation):", 0) print() print("rule points opened listening service reached pair open points") open_points = listening = 0 for ad, points, listens in RULE: open_points += points listening += listens print(f" {ad:10s} {points:12d} {listens:16d} {listening:14d} {open_points:11d}") print(f" {'total':10s} {open_points:12d} {listening:16d} {listening:14d} {open_points:11d}") print(" surface that can grow without the rule changing:", open_points - listening, "pairs") print() ORACLE = "internal-host-07" CANDIDATE = tuple(f"internal-host-{i:02d}" for i in range(1, INTERNAL_HOSTS + 1)) def log_line(source, translated): return "gateway" if translated else source def translation_table(source, readable): return source if readable else "no-record" print("reading distinct answers eliminated remaining") for ad, f in (("target's log (with translation)", lambda k: log_line(k, True)), ("target's log (without translation)", lambda k: log_line(k, False)), ("translation table (while the record is alive)", lambda k: translation_table(k, True)), ("translation table (after the record drops)", lambda k: translation_table(k, False))): answer = f(ORACLE) remaining = sum(1 for a in CANDIDATE if f(a) == answer) print(f" {ad:38s} {len({f(a) for a in CANDIDATE}):5d} {len(CANDIDATE) - remaining:7d}" f" {remaining:6d}") print() CONNECTION_LIST = [(15 * i, 30 + (i % 7) * 10) for i in range(CONNECTIONS)] print(f"translation entry: {CONNECTIONS} connections, {WINDOW}-second window," f" the record drops {TIMEOUT} seconds after closing") print(" look time started still resolvable ratio") for T in (0, 100, 300, 600): started = [(b, s) for b, s in CONNECTION_LIST if b <= T] resolvable = [1 for b, s in started if b + s + TIMEOUT > T] ratio = round(len(resolvable) / len(started), 4) if started else 0.0 print(f" {T:9d} {len(started):9d} {len(resolvable):17d} {ratio:9.4f}") print() print("candidate set sweep: internal host count changes") for n in (8, 12, 16): a = tuple(f"internal-host-{i:02d}" for i in range(1, n + 1)) remaining = sum(1 for x in a if log_line(x, True) == "gateway") print(f" {n:2d} internal hosts -> translated log eliminated {len(a) - remaining:2d}" f" remaining {remaining:2d} | untranslated log eliminated {len(a) - 1:2d}" f" remaining {1:2d}") ``` ``` internal network: 12 hosts x 5 services = 60 internal (host, service) pairs pairs directly reachable from outside (no translation): 0 rule points opened listening service reached pair open points single-a 1 1 1 1 single-b 1 1 2 2 single-c 1 1 3 3 range-d 10 5 8 13 total 13 8 8 13 surface that can grow without the rule changing: 5 pairs reading distinct answers eliminated remaining target's log (with translation) 1 0 12 target's log (without translation) 12 11 1 translation table (while the record is alive) 12 11 1 translation table (after the record drops) 1 0 12 translation entry: 40 connections, 600-second window, the record drops 120 seconds after closing look time started still resolvable ratio 0 1 1 1.0000 100 7 7 1.0000 300 21 12 0.5714 600 40 11 0.2750 candidate set sweep: internal host count changes 8 internal hosts -> translated log eliminated 0 remaining 8 | untranslated log eliminated 7 remaining 1 12 internal hosts -> translated log eliminated 0 remaining 12 | untranslated log eliminated 11 remaining 1 16 internal hosts -> translated log eliminated 0 remaining 16 | untranslated log eliminated 15 remaining 1 ``` #### The Opened Surface The table above counts, point by point, the surface the four rules open. With no translation, the internal pairs reachable from outside are **0**; none of the sixty pairs in the internal network can be reached. The three single-point rules raise this to three, the fourth rule opens a ten-point range, and the pairs reached today come to **8**. The gap between the two numbers is this section's real subject. Open points: **13**. Pairs reached: **8**. The **5** pairs in between are closed today because no service listens on those points; the day a service starts on that host, they become reachable **without the rule changing**. A rule written as a range does not define today's surface — it defines the **largest future surface**. The number that must be measured is not eight, it is thirteen. This ties the choice between a single-point rule and a range rule to a single number. When the same job is written as five single-point rules instead of one range rule, the open-point count drops from thirteen to eight, and today's function does not change. #### Three Numbers **Oracle:** the true source, `internal-host-07` — one of twelve internal hosts. **Test:** the target server's log writes a single source field, and with translation in place, the gateway's address sits there; for twelve hosts, **a single answer**. **Eliminated candidates:** translated log **0**, untranslated log **11**, translation table while the record is alive **11**, after the record drops **0**. This zero is the course's rule in its shortest form. The log line exists, it is readable, it is correct, and it **eliminates no candidate**. Source translation reduces twelve distinct identities to a single one; every line on the target side says the same thing. A record's existing is not enough to count as a measurement; **output whose diagnostic power is not tested counts as unmeasured.** The bottom table shows the sweep. As the internal host count rises from eight to sixteen, the candidates the translated log eliminates stay fixed at **0**, while the remaining count rises from eight to sixteen. A reading with zero elimination power grows even less useful as the candidate set grows. #### The Record's Lifetime The only place that can bring identity back is the translation table, and that table is a memory structure, not a log. A record lives while the connection is open and drops one hundred twenty seconds after it closes. Looked at the hundredth second, **seven** of the seven connections that had started are resolvable; the ratio is **1.0000**. At the three-hundredth second, **twelve** of twenty-one; the ratio is **0.5714**. At the six-hundredth second, only **eleven** of forty; the ratio is **0.2750**. An operator looking with a ten-minute delay cannot recover the source of three-quarters of the connections from anywhere. **Evidence does not wait.** The only way to keep the source permanently is to write a log line at the moment of translation, and that is a setting; it must be turned on before the fault. When it is not on, what remains is the target's log, and that log eliminates zero of twelve candidates. #### Summary - Source translation runs at the stop after the routing decision, destination translation at the stop before it; filtering rules see the packet in its translated form, and a rule written against the external address does not match. - Four forwarding rules open **13** points, and today **8** pairs are reached; the **5** pairs in between can open in the future without the rule changing. A range rule defines not today's surface but the largest future surface. - The translated log line eliminates **0** of twelve candidate sources; the untranslated log and a live translation entry eliminate **11**. - A translation entry drops 120 seconds after closing: resolvability is **1.0000** at the 100th second, **0.5714** at 300, **0.2750** at 600. - When the internal host count rises from eight to sixteen, the candidates the translated log eliminates stay fixed at **0**, and the remaining count rises from eight to sixteen. #### Course Wrap-Up This course asked one question thirteen times: **how many candidate causes does a test eliminate.** In every lesson three numbers stood side by side — the oracle, the answer the test gives, and the count of eliminated candidate causes. The course's rule was this: a test's number is not the answer it gives but the candidate causes it eliminates; **a test whose elimination count is not written down counts as unmeasured.** The table below gathers the measure of all thirteen lessons in one place. **No row left blank has been made up**: each row is written only from its own source lesson. | Lesson | Oracle | Test | Eliminated candidates / cost | |---|---|---|---| | Interfaces and Addresses | 8 candidate causes, 8 tests | interface test **3 distinct answers** | **6 candidates** remain in the worst case | | The Routing Table | same 8 candidates | route test **3 answers** | worst case **5 candidates**; `wrong-gateway` and `missing-route` are not separated | | Name Resolution | same 8 candidates | name resolution **2 answers** | `unresolved` eliminates **7**, `resolved` only **1** — the course's largest answer gap | | Ports and Listening Services | same 8 candidates | listening list **2 answers** | only **1** candidate on the expected answer; connection attempt **7** at once | | The Layered Diagnosis Method | `firewall-dropping` | connection attempt `timeout` | **7 eliminated, 1 remaining**; layered **33**, reverse **38**, greedy **10** (unreachable lower bound) | | Reachability Tests | `wrong-gateway` | remote echo `no-response` | **4 eliminated, 4 remaining**; the echo **responding** also leaves 4 candidates standing | | Packet Capture | `firewall-dropping` | unfiltered capture **14,400 lines** | **6 eliminated, 2 remaining**; elimination per line **0.0004** against **7.0000** for the connection attempt | | Common Fault Patterns | `resolver-silent` | name resolution `unresolved` | **7 eliminated, 1 remaining**; intermittent fault 84 s, capture **42/84** at 300, **13/84** at 500 | | Secure Shell | 9 candidates, `public-key-unauthorized` | verbose client 46 lines, verbose server log 3 lines | **5 / 7**; the listening list eliminates **1** in 12 lines; seven tests leave **2** indistinguishable groups | | Tunneling and the Jump Server | 10 candidates, `target-firewall-dropping` | end-to-end **7** distinct answers, jump-to-target **5** | **8 / 9**; layered order **34**, end-to-end-first **28** steps; tunnel entry point **0** and **72** indirect pairs | | The Packet-Filtering Framework | the decision is made by **policy** at the `input-filter` stop | rule list **34** lines, open tracing **12** lines | **0 / 8**; loaded counter **0**, quiet counter **6**; unreached hops **20** | | Firewall Rules | **40** of 120 orders lock out, the final set is the same across all | dry run **6** answers, end-to-end attempt **4** answers | **5 / 5**, grouped together **0**; dry run prevents **0** lockouts, atomic load **40** | | Address Translation | true source `internal-host-07` | translated log **1** distinct answer | **0**, remaining **12**; resolvability at the 600th second **0.2750** | The five lessons' shared result is a single pattern. At the access layer there is no link between the volume of output and its elimination power: the forty-six-line verbose client output eliminated five candidates, the three-line server log eliminated seven, the thirty-four-line rule list eliminated none, and the one-line translated log also eliminated none. Elimination power is not in the line count, it is in **how many distinct answers** the output can produce. The second pattern concerned the candidate set. In every lesson the list was grown and shrunk by one candidate, and the result pointed the same way each time: the elimination count stayed fixed while the remaining ambiguity grew. This is why, throughout this topic, the measure used was not elimination but **remaining** and **indistinguishable group**. The third pattern concerned time, and it took its harshest form here. With tracing off, a dropped packet's trace was zero lines; the log a locked-out administrator would need to read was inside the machine; the translation entries had fallen to eleven of forty by the six-hundredth second. In all three, the evidence depended on a setting, and that setting had to be turned on before the fault. **Evidence does not wait.** Throughout this topic, destructive and lockout-causing commands were never written in full form, no real key or fingerprint was written, and address translation was explained not as "open this" but as **how many pairs get opened**. This was not a teaching restriction, it was the measurement itself: an open surface is a countable quantity. The next course, **Kernel Interfaces and Isolation**, enters the place this course deliberately left out. Here, a machine had a single network stack; there, namespaces set up multiple network stacks on the same machine, and every interface, every routing table, and every rule chain from this course **multiplies**. Container networking, namespaces, and control groups were not covered in this course and were left for there. The same question will be asked there too: how many candidate causes does a test eliminate. What makes the answer harder is that the candidate set is now split into multiple copies even inside a single machine.