Tunneling and the Jump Server

The previous lesson’s sharpest result was this: all the elimination power sits on the server, and the way onto the server is closed. The four tests that can be done from outside left a four-member group unseparated. The usual solution to this blockage is going to a machine that cannot be reached directly, by way of another machine that can be reached.

The solution comes with a cost, and this lesson counts that cost. When the path splits in two, the candidate-cause set splits in two as well: the fault can now be on either of the two machines, on either of the two legs in between, or on the client itself. How much of this expanded set a single end-to-end attempt eliminates is the question.

The Path’s Two Stops

A jump host is the machine that stands as the sole gate between two networks. The client connects to it, and it connects to the target. There are two arrangements, and they produce different candidate causes.

In the first, a local port is opened on the client machine, and traffic arriving at that port is carried through the encrypted session and forwarded to the target. This is called a tunnel. In the second, the jump host is only a carrier: the session opens directly on the target, and no shell runs on the jump host. The dump below is not executed; it is meant to show the syntax of the two forms:

# example dump, not executed
# local forwarding: the mouth opens on the client, traffic goes to the target via the jump
ssh -L 127.0.0.1:<local-port>:<target-host>:<target-port> <user>@<jump>

# using as a hop: the session opens on the target, the jump only carries
ssh -J <user>@<jump> <user>@<target-host>

The detail shared by both forms decides half the diagnosis: the <target-host> name is resolved on the jump host, not on the client. These two forms work only if the target’s name is defined in the remote network’s resolver. If the same name is written as an alias in the client’s own configuration, the client tries to resolve it and cannot. The resulting error message points at name resolution, even though the name resolves perfectly well on the server. Which side a name resolves on is this topic’s own particular candidate cause.

How a jump host narrows the number of directly reachable machines was counted in the remote-access hardening lesson of the DevOps and Platform Engineering curriculum, and its placement between trust zones in the perimeter defense topic of the Cybersecurity curriculum. What is measured here is different: what the same arrangement does to diagnosis.

Where the Tunnel’s Mouth Binds

The tunnel opens a port on the client machine, and that port’s bind address is a choice. Bound to the loopback address, only processes on that machine can use the tunnel. Bound to all interfaces, every host on the client’s network segment can use it too — and at the other end of the tunnel sits a target those hosts cannot reach directly.

This is a countable growth of exposed surface. In the setup, 24 other hosts sit on the client’s segment, and 3 tunnels are open. With tunnels bound to loopback, the number of other hosts that can reach the tunnel’s mouth is 0, and the number of (host, target service) pairs that become indirectly reachable is 0. Bound to all interfaces, these two numbers become 24 and 72. The difference is a single configuration field, and a tunnel written without knowing its default opens a path thought closed to twenty-four hosts.

Ten Candidate Causes

The setup is this: the client is trying to reach a service on the target through the jump host and cannot. There are ten candidate causes, each breaking a single field of the healthy state.

• EF11 — The candidate-cause list has ten items and is exhaustive. Two are on the client, two on the first leg, one in the jump host’s configuration, four on the target, one in name resolution’s location.
• EF12 — The oracle is chosen as target-firewall-dropping: the target’s filter silently drops the packet.
• EF13 — Six tests are defined. Three can be done on the client, two require opening a session on the jump host, one compares the two sides.
• EF14 — Tests that run on the jump host cannot be done if the first leg is broken, and they report this with a separate answer; this is a dependency and is part of the measurement.
• EF15 — The end-to-end attempt covers opening the tunnel and a session attempt through it together; for this reason it can also see the authentication result on the target.
• EF16 — The layered order proceeds from client to target; the second order puts the end-to-end attempt first. Both can be applied without knowing the oracle.
• EF17 — The tunnel entry point measurement assumes 24 other hosts and 3 open tunnels in the segment.
• EF18 — The candidate set sweep is done with nine-, ten-, and eleven-candidate lists.
• EF19 — jump-session-restricted, added to the eleven-candidate list, is the jump host blocking forwarding at the session level, and it gives the same answer as forwarding-disabled.
• EF20 — All counts are exhaustive; there is no randomness or seed.

"""Two-hop path: client -> jump host -> target. Ten candidates, six tests."""
CANDIDATE = ("local-port-busy", "local-address-wrong", "jump-unreachable",
        "jump-auth-refused", "forwarding-disabled", "target-unreachable",
        "target-firewall-dropping", "target-not-listening", "target-auth-refused",
        "name-resolves-remote-only")
HEALTHY = {"local_bound": True, "local_address_correct": True, "jump_reachable": True,
          "jump_auth": True, "forwarding_allowed": True, "target_reachable": True,
          "target_firewall_allows": True, "target_listening": True,
          "target_auth": True, "name_resolves_locally": True}
FAULT = {
    "local-port-busy": {"local_bound": False},
    "local-address-wrong": {"local_address_correct": False},
    "jump-unreachable": {"jump_reachable": False},
    "jump-auth-refused": {"jump_auth": False},
    "forwarding-disabled": {"forwarding_allowed": False},
    "target-unreachable": {"target_reachable": False},
    "target-firewall-dropping": {"target_firewall_allows": False},
    "target-not-listening": {"target_listening": False},
    "target-auth-refused": {"target_auth": False},
    "name-resolves-remote-only": {"name_resolves_locally": False},
}


def state(a=None):
    d = dict(HEALTHY)
    if a:
        d.update(FAULT[a])
    return d


def test_end_to_end(d):
    if not d["local_bound"] or not d["local_address_correct"]:
        return "local-connection-refused"
    if not d["name_resolves_locally"]:
        return "name-unresolved"
    if not d["jump_reachable"] or not d["jump_auth"]:
        return "hop-not-opened"
    if not d["forwarding_allowed"]:
        return "channel-forbidden"
    if not d["target_reachable"] or not d["target_firewall_allows"]:
        return "channel-timeout"
    if not d["target_listening"]:
        return "channel-refused"
    return "established" if d["target_auth"] else "permission-denied-at-target"


def test_jump_session(d):
    if not d["jump_reachable"]:
        return "unreachable"
    return "opened" if d["jump_auth"] else "permission-denied"


def test_local_listener(d):
    if not d["local_bound"]:
        return "not-bound"
    return "expected-address" if d["local_address_correct"] else "different-address"


def test_jump_to_target(d):
    if not d["jump_reachable"] or not d["jump_auth"]:
        return "could-not-be-done"
    if not d["target_reachable"]:
        return "unreachable"
    if not d["target_firewall_allows"]:
        return "timeout"
    return "established" if d["target_listening"] else "refused"


def test_forwarding_setting(d):
    if not d["jump_reachable"] or not d["jump_auth"]:
        return "unreadable"
    return "allowed" if d["forwarding_allowed"] else "disabled"


TEST = {"end-to-end": test_end_to_end, "local-listener": test_local_listener,
          "jump-session": test_jump_session,
          "forwarding-setting": test_forwarding_setting,
          "jump-to-target": test_jump_to_target,
          "name-location": lambda d: ("both-sides" if d["name_resolves_locally"]
                                else "remote-only")}
LAYERED = ["local-listener", "name-location", "jump-session", "forwarding-setting",
            "jump-to-target", "end-to-end"]
END_TO_END_FIRST = ["end-to-end"] + LAYERED[:-1]


def eliminate(candidates, s, answer):
    return tuple(a for a in candidates if TEST[s](state(a)) == answer)


def group(tests, candidates=CANDIDATE):
    o = {}
    for a in candidates:
        o.setdefault(tuple(TEST[s](state(a)) for s in tests), []).append(a)
    return [sorted(v) for v in o.values() if len(v) > 1]


def order_steps(order, real_fault, candidates=CANDIDATE):
    remaining, step = tuple(candidates), 0
    for s in order:
        if len(remaining) <= 1:
            break
        step += 1
        remaining = eliminate(remaining, s, TEST[s](state(real_fault)))
    return step, len(remaining)


ORACLE = "target-firewall-dropping"
print("candidate causes:", len(CANDIDATE), "| tests:", len(TEST), "| oracle:", ORACLE)
print()
print("test                distinct answers  worst-case remaining  eliminated  remaining")
for s in TEST:
    o = {}
    for a in CANDIDATE:
        o.setdefault(TEST[s](state(a)), []).append(a)
    remaining = len(eliminate(CANDIDATE, s, TEST[s](state(ORACLE))))
    print(f"  {s:18s} {len(o):11d} {max(len(v) for v in o.values()):14d}"
          f" {len(CANDIDATE) - remaining:7d} {remaining:6d}")
print()
print("fault                        layered  end-to-end-first  end-to-end alone remaining")
tk = tu = 0
for a in CANDIDATE:
    k, _ = order_steps(LAYERED, a)
    u, _ = order_steps(END_TO_END_FIRST, a)
    _, y = order_steps(["end-to-end"], a)
    tk += k
    tu += u
    print(f"  {a:26s} {k:8d} {u:11d} {y:22d}")
print(f"  {'total':26s} {tk:8d} {tu:11d}")
print()
print("tool set                          indistinguishable group")
for ad, k in (("end-to-end alone", ["end-to-end"]),
              ("end-to-end + jump session", ["end-to-end", "jump-session"]),
              ("end-to-end + jump-to-target",
               ["end-to-end", "jump-to-target"]),
              ("three tests running locally",
               ["end-to-end", "local-listener", "name-location"]),
              ("six tests", list(TEST))):
    o = group(k)
    print(f"  {ad:35s} {len(o):8d}", o if o else "")
print()
NEIGHBORS, TUNNELS = 24, 3
print(f"the tunnel entry point's bind address ({NEIGHBORS} other hosts in the subnet, {TUNNELS} tunnels open)")
for ad, reachers in (("loopback", 0), ("all interfaces", NEIGHBORS)):
    print(f"  {ad:14s} other hosts that can reach the tunnel entry {reachers:3d}"
          f" | indirect (host, target service) pairs {reachers * TUNNELS:3d}")
print()
print("candidate set sweep")
FAULT["jump-session-restricted"] = {"forwarding_allowed": False}
for ad, k in (("9 candidates: target-auth-refused removed",
               tuple(a for a in CANDIDATE if a != "target-auth-refused")),
              ("10 candidates: base list", CANDIDATE),
              ("11 candidates: jump-session-restricted added",
               CANDIDATE + ("jump-session-restricted",))):
    remaining = len(eliminate(k, "end-to-end", TEST["end-to-end"](state(ORACLE))))
    o = group(list(TEST), k)
    print(f"  {ad:42s} end-to-end eliminated {len(k) - remaining:2d} remaining {remaining:2d}"
          f" | indistinguishable group {len(o)}"
          f" largest {max((len(x) for x in o), default=0)}")

candidate causes: 10 | tests: 6 | oracle: target-firewall-dropping

test                distinct answers  worst-case remaining  eliminated  remaining
  end-to-end                   7              2       8      2
  local-listener               3              8       2      8
  jump-session                 3              8       2      8
  forwarding-setting           3              7       3      7
  jump-to-target               5              5       9      1
  name-location                2              9       1      9

fault                        layered  end-to-end-first  end-to-end alone remaining
  local-port-busy                   1           2                      2
  local-address-wrong               1           2                      2
  jump-unreachable                  3           4                      2
  jump-auth-refused                 3           4                      2
  forwarding-disabled               4           1                      1
  target-unreachable                5           6                      2
  target-firewall-dropping          5           6                      2
  target-not-listening              5           1                      1
  target-auth-refused               5           1                      1
  name-resolves-remote-only         2           1                      1
  total                            34          28

tool set                          indistinguishable group
  end-to-end alone                           3 [['local-address-wrong', 'local-port-busy'], ['jump-auth-refused', 'jump-unreachable'], ['target-firewall-dropping', 'target-unreachable']]
  end-to-end + jump session                  2 [['local-address-wrong', 'local-port-busy'], ['target-firewall-dropping', 'target-unreachable']]
  end-to-end + jump-to-target                2 [['local-address-wrong', 'local-port-busy'], ['jump-auth-refused', 'jump-unreachable']]
  three tests running locally                2 [['jump-auth-refused', 'jump-unreachable'], ['target-firewall-dropping', 'target-unreachable']]
  six tests                                  0 

the tunnel entry point's bind address (24 other hosts in the subnet, 3 tunnels open)
  loopback       other hosts that can reach the tunnel entry   0 | indirect (host, target service) pairs   0
  all interfaces other hosts that can reach the tunnel entry  24 | indirect (host, target service) pairs  72

candidate set sweep
  9 candidates: target-auth-refused removed  end-to-end eliminated  7 remaining  2 | indistinguishable group 0 largest 0
  10 candidates: base list                   end-to-end eliminated  8 remaining  2 | indistinguishable group 0 largest 0
  11 candidates: jump-session-restricted added end-to-end eliminated  9 remaining  2 | indistinguishable group 1 largest 2

Three Numbers

Oracle: the real cause is target-firewall-dropping — the target’s filter silently drops the packet. Test: the end-to-end attempt says “channel timed out”; the attempt made from the jump host to the target says “timeout”; the local listener list says the tunnel is bound to the expected address. Candidates eliminated: end-to-end 8, hop-to-target 9, local listener 2, name location 1.

The end-to-end attempt is the strongest single test in this setup: it produces seven distinct answers and, in the worst case, leaves only two candidates. Even so, it cannot find the oracle on its own, because not being able to reach the target and the target’s filter dropping the packet look the same from inside the tunnel. What separates them is the attempt made from the jump host: there, “unreachable” and “timeout” are different answers, and that test alone eliminates nine candidates.

The placement rule that follows is this: a jump host is as much an observation point as it is a gate. It sits at the point nearest the target, and an attempt made there removes every leg in between from the equation. Being able to open a session on a jump host means a nine-candidate elimination in diagnosis.

Order and Indistinguishable Groups

The second table compares two fixed orders. The layered order proceeds from client to target and spends a total of 34 steps for the ten faults. The order that puts the end-to-end attempt first finishes in 28 steps. Both can be applied without knowing the oracle; the six-step difference between them comes from the end-to-end attempt producing seven distinct answers.

This result does not contradict what was measured on a single machine. There, the connection attempt gave four distinct answers; here, the attempt made through the tunnel rises to seven, because the tunnel carries back the failure at each stop along the path as a separate message. A test’s power is not in the tool’s name, it is in how many distinct answers it produces.

The third table shows where the elimination power runs out. Looking only at end-to-end, three groups cannot be separated: two local faults, two first-leg faults, and two silent target faults. Adding the jump session resolves the second group, adding the hop-to-target attempt the third. The three local tests, on the other hand, leave two groups; none of them can get past the client. When all six tests are used together, the indistinguishable group count is 0.

Candidate Set Sweep

The bottom table tries the candidate list at three sizes. When target-auth-refused is removed, the number of candidates the end-to-end attempt eliminates drops from eight to seven, and what remains is still two. When jump-session-restricted is added to the list, the eliminated count rises to nine and what remains is still two; against this, a group that cannot be separated even with all six tests is born, because the new candidate gives the same answer as forwarding-disabled on every test.

One detail falls outside this measurement and needs to be stated. The tunnel’s channel error appears once, on the client’s terminal, and is written nowhere. If the tunnel was opened in the background, that single line is not seen either; what remains is only a connection that does not work. The record of the session at the hop is in the jump host’s log, and whether the channel opening and closing is recorded there depends on the verbosity level set there. The evidence does not wait: looking for the channel error afterward gives a result only if the verbosity level was sufficient at the moment of the fault.

These three lines show the previous lesson’s rule from another direction. There, the elimination count stayed fixed while what remained grew; here, as the elimination count grows, distinguishability drops. In both cases, elimination on its own is not a measure of success. A candidate set is only considered resolved if the tests’ answer patterns can separate it one by one, and the measure of this is the count of indistinguishable groups.

Summary

• A jump host splits the path in two: in tunnel form a mouth opens on the client, in hop form the session opens directly on the target and the jump host only carries.
• The target’s name resolves on the jump host; if the same name is written in the client’s configuration, the client tries to resolve it and a name resolution fault that does not exist on the server appears.
• The tunnel entry point’s bind address determines the exposed surface: at loopback, indirectly reachable pairs are 0; at all interfaces, for 24 hosts and 3 tunnels in the subnet, 72.
• Across ten candidate causes, the end-to-end attempt gives seven distinct answers and eliminates 8 candidates; the attempt made from the jump host to the target eliminates 9 and resolves it on its own; the location of name resolution eliminates only 1.
• The layered order spends 34 steps, the order that puts the end-to-end attempt first spends 28; both can be applied without knowing the oracle.
• When one item is added to the candidate list, the eliminated count rises from eight to nine while the indistinguishable group count rises from zero to one; the measure is not elimination but distinguishability.

Next Step

This lesson’s oracle was the target’s filter, and looked at from inside the tunnel, that filter could not be told apart from not reaching the target at all. A silently dropped packet is the hardest event to trace in this course, because it sends nothing back. The next lesson goes inside that filter: it builds which stops a packet passes through inside a machine, in what order rules are evaluated, and which reading can show at which stop a packet was dropped.