Secure Shell

The previous topic chased a fault’s trace and showed that the trace is often not left for someone looking late: an intermittent fault begins before the recording starts, a dropped packet writes no line by default. This topic asks the same question of access itself. When the way to connect to a server closes, a single line sits in front of you, and that line usually says “permission denied.”

The question to ask is how many candidate causes that line eliminates. The course’s rule does not change — a test’s number is not the answer it gives, but what it eliminates. In this lesson, even when the tests are put together, a pair will remain that they cannot eliminate, and what separates that pair is not a tool but a setting made before the fault.

The Two Ends of the Key Pair

Two separate authentications stand stacked in a secure shell connection, and confusing them breaks the diagnosis. In the first, the client authenticates the server: the server’s host key is recorded on the first connection and compared against the record on later connections. In the second, the server authenticates the user: the user’s private key stays on the client, and its counterpart, the public key, is found in the authorized keys file on the server.

The key pair is generated on the client, and the private key never leaves it; only the public key is copied to the server. This asymmetry is the basis of hardening: even if the server is compromised, what sits there is not enough to produce the user’s identity. No real key, fingerprint, or password is written in this lesson; wherever a key string would be, a placeholder in square brackets is used.

Server-side hardening consists of a few settings. The dump below is not executed; it is an example written to show the settings’ names and the values they take:

# example dump, not executed
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
LogLevel VERBOSE

Each of these lines changes the candidate-cause set, and this is the measurement’s real subject. Turning off password authentication removes every password-related candidate from the list and puts one in its place: no user without a key is left a fallback method. Fixing the location of the authorized keys file removes the candidate of the file being looked for in the wrong place. Raising the log’s verbosity level removes no candidate, but it increases distinguishability, and that is exactly what this lesson’s final measure counts.

There is one more setting, and it is often overlooked: if the permissions of the authorized keys file and the directory containing it are too open, the server will not read the file. The file is in place, its content is correct, the key would match; the server still refuses. This behavior adds a separate candidate cause to the list, and it is the candidate that produces the most frustrating result in the measurement.

The Public Key Crossing to the Other Side

Once the key pair is generated, one task remains: appending the public key to the correct user’s authorized keys file on the server. This task produces more candidate causes than it looks like it should, because it can go wrong in four separate places. The key can be appended to the wrong user’s file; the file can be correct but merge with the previous line because a line ending is missing; if the directory does not exist on the server at all, the append can silently fall to a different path; a space per line entered during copying can invalidate the key. What the four share is that the operation’s output comes back successful.

For this reason, the key being placed is not verified on its own. Verification is done not by the operation’s output but by a new session attempt, and the attempt is made without closing the current session. This detail will be repeated exactly the same way in rule writing in later lessons: what verifies a change is not the value returned by the command that made the change, but an attempt made through a separate channel.

The key’s lifetime, its regular rotation, being stored in a central vault, and being signed with a certificate are not this course’s subject; these are measured in the key management lessons of the Cybersecurity curriculum. What is measured here is only how much a single machine’s local configuration makes distinguishable in the face of a fault.

Nine Candidate Causes

The setup is this: the user is trying to connect and the session does not open. There are nine candidate causes, each breaking a single field of the healthy state. The shared reference’s eight network faults are gathered here into a single candidate — the network being unreachable; this topic measures the layer above the network.

• EF1 — The candidate-cause list has nine items and is exhaustive; the setup never goes outside these nine states.
• EF2 — The oracle is chosen as public-key-unauthorized: the user has a key, and it has no counterpart in the server’s authorized keys file.
• EF3 — Seven tests are defined. Four can be done from outside, without opening a session; two require running on the server, and one is the server log’s basic level.
• EF4 — Each test is given a line count. This number is part of the setup, not a measured value; its purpose is to place output volume and elimination power side by side.
• EF5 — The verbose client output is counted as 46 lines, the verbose server log 3, the basic server log 1, the listening list 12, the local key check 4, the method list 9, the default client message 1.
• EF6 — Verbose logging is assumed to have been turned on before the fault; turning it on afterward does not bring back a past attempt.
• EF7 — The breadth of the key permissions and the public key being unauthorized fall into the same failure branch on the server; this reflects real behavior, not the setup.
• EF8 — The candidate set sweep is done with three lists: eight, nine, and ten candidates.
• EF9 — method-disabled, added to the ten-candidate list, is the case of the server rejecting the key method, and it gives the same answer pattern as public-key-unauthorized.
• EF10 — All counts are exhaustive; there is no randomness or seed.

"""Secure shell session does not open: nine candidate causes, seven tests.

Oracle = the real cause (we chose the setup). What is measured is how many
candidates each test eliminates and how many remain.
"""
CANDIDATE = ("network-unreachable", "service-not-listening", "host-key-changed",
        "private-key-missing", "public-key-unauthorized", "key-permissions-too-open",
        "user-missing", "account-locked", "password-method-disabled")
HEALTHY = {"transport_established": True, "service_listening": True,
          "host_key_matches": True, "client_offers_key": True,
          "server_accepts": True, "account_valid": True, "password_open": True}
FAULT = {
    "network-unreachable": {"transport_established": False},
    "service-not-listening": {"transport_established": False, "service_listening": False},
    "host-key-changed": {"host_key_matches": False},
    "private-key-missing": {"client_offers_key": False, "server_accepts": False},
    "public-key-unauthorized": {"server_accepts": False},
    "key-permissions-too-open": {"server_accepts": False},
    "user-missing": {"server_accepts": False, "account_valid": False},
    "account-locked": {"server_accepts": False, "account_valid": False},
    "password-method-disabled": {"password_open": False},
}
LINES = {"client-message": 1, "verbose-client": 46, "local-key": 4,
         "method-list": 9, "listening": 12, "log-basic": 1,
         "log-verbose": 3}


def state(a=None):
    d = dict(HEALTHY)
    if a:
        d.update(FAULT[a])
    return d


def test_client(d):
    if not d["transport_established"]:
        return "no-connection"
    if not d["host_key_matches"]:
        return "host-key-warning"
    return "session-opened" if d["server_accepts"] else "permission-denied"


def test_verbose_client(d):
    if not d["transport_established"]:
        return "transport-failed"
    if not d["host_key_matches"]:
        return "host-key-mismatch"
    if not d["client_offers_key"]:
        return "key-not-offered"
    return "session-opened" if d["server_accepts"] else "key-rejected"


def test_log_basic(d):
    if not d["transport_established"]:
        return "no-line"
    return "accepted" if d["server_accepts"] else "failed"


def test_log_verbose(d):
    if not d["transport_established"]:
        return "no-line"
    if not d["account_valid"]:
        return "invalid-account"
    if not d["client_offers_key"]:
        return "method-not-offered"
    return "accepted" if d["server_accepts"] else "key-mismatch"


TEST = {
    "client-message": test_client, "verbose-client": test_verbose_client,
    "local-key": lambda d: ("key-present" if d["client_offers_key"]
                                else "key-absent"),
    "method-list": lambda d: ("password-open" if d["password_open"]
                                 else "password-closed"),
    "listening": lambda d: "listening" if d["service_listening"] else "not-listening",
    "log-basic": test_log_basic, "log-verbose": test_log_verbose,
}
FROM_OUTSIDE = ["client-message", "verbose-client", "local-key", "method-list"]
ON_SERVER = ["listening", "log-verbose"]
ORACLE = "public-key-unauthorized"


def eliminate(candidates, s, answer):
    return tuple(a for a in candidates if TEST[s](state(a)) == answer)


def group(tests, candidates=CANDIDATE):
    o = {}
    for a in candidates:
        o.setdefault(tuple(TEST[s](state(a)) for s in tests), []).append(a)
    return [sorted(v) for v in o.values() if len(v) > 1]


print("candidate causes:", len(CANDIDATE), "| tests:", len(TEST), "| oracle:", ORACLE)
print()
print("test                lines  distinct answers  eliminated  remaining  per line")
for s in TEST:
    remaining = len(eliminate(CANDIDATE, s, TEST[s](state(ORACLE))))
    el = len(CANDIDATE) - remaining
    print(f"  {s:18s} {LINES[s]:5d} {len({TEST[s](state(a)) for a in CANDIDATE}):11d}"
          f" {el:7d} {remaining:6d} {el / LINES[s]:13.4f}")
print()
print("tool set                       lines  indistinguishable groups  largest group")
for ad, k in (("client message alone", ["client-message"]),
              ("verbose client alone", ["verbose-client"]),
              ("4 accessible from outside", FROM_OUTSIDE),
              ("outside 4 + basic log", FROM_OUTSIDE + ["log-basic"]),
              ("2 running on server", ON_SERVER),
              ("seven tests", list(TEST))):
    o = group(k)
    print(f"  {ad:28s} {sum(LINES[s] for s in k):5d} {len(o):17d}"
          f" {max((len(x) for x in o), default=0):14d}")
print()
print("candidate set sweep (same oracle)")
FAULT["method-disabled"] = {"server_accepts": False}
for ad, k in (("8 candidates: account-locked removed",
               tuple(a for a in CANDIDATE if a != "account-locked")),
              ("9 candidates: base list", CANDIDATE),
              ("10 candidates: method-disabled added",
               CANDIDATE + ("method-disabled",))):
    remaining = len(eliminate(k, "log-verbose",
                    TEST["log-verbose"](state(ORACLE))))
    o = group(list(TEST), k)
    print(f"  {ad:35s} verbose log eliminated {len(k) - remaining:2d}"
          f" remaining {remaining:2d} | indistinguishable group {len(o)}"
          f" largest {max((len(x) for x in o), default=0)}")
print()
print("indistinguishable groups (seven tests):", group(list(TEST)))

candidate causes: 9 | tests: 7 | oracle: public-key-unauthorized

test                lines  distinct answers  eliminated  remaining  per line
  client-message         1           4       4      5        4.0000
  verbose-client        46           5       5      4        0.1087
  local-key              4           2       1      8        0.2500
  method-list            9           2       1      8        0.1111
  listening             12           2       1      8        0.0833
  log-basic              1           3       4      5        4.0000
  log-verbose            3           5       7      2        2.3333

tool set                       lines  indistinguishable groups  largest group
  client message alone             1                 2              5
  verbose client alone            46                 2              4
  4 accessible from outside       60                 2              4
  outside 4 + basic log           61                 2              4
  2 running on server             15                 3              2
  seven tests                     76                 2              2

candidate set sweep (same oracle)
  8 candidates: account-locked removed verbose log eliminated  6 remaining  2 | indistinguishable group 1 largest 2
  9 candidates: base list             verbose log eliminated  7 remaining  2 | indistinguishable group 2 largest 2
  10 candidates: method-disabled added verbose log eliminated  7 remaining  3 | indistinguishable group 2 largest 3

indistinguishable groups (seven tests): [['key-permissions-too-open', 'public-key-unauthorized'], ['account-locked', 'user-missing']]

Three Numbers

Oracle: the real cause is public-key-unauthorized — the key is offered, the server does not recognize it. Test: the default client message says “permission denied” in a single line; the verbose client output prints forty-six lines and says “key rejected”; the verbose server log says “key mismatch” in three lines. Candidates eliminated: the default message 4, the verbose client 5, the verbose server log 7.

The comparison in the middle is this lesson’s form of the course’s second claim. The verbose client output produces fifteen times as many lines as the server log and eliminates two fewer candidates. The per-line elimination ratio is 0.1087 against 2.3333 — a factor of twenty-one apart. The reason is structural: the verbose client output details the client side of the handshake, while every one of the unseparated candidates is on the server side. Raising the verbosity level enlarges the side that is visible; it does not open up the side that is not.

The same table’s weakest row is listening. It prints twelve lines and eliminates only one candidate. In the shared reference, the listening list’s elimination power was also one; on the access layer, it is one as well. Seeing that the service is listening does not show that you can connect, and the distance between these two sentences is this entire course.

The Pair That Cannot Be Separated

The bottom table says something harsher. Even when all seven tests are used together, two groups remain unseparated, and both have two members: public-key-unauthorized with key-permissions-too-open, account-locked with user-missing. Seventy-six lines of output are produced, and these four states cannot be split two and two.

The first pair ties back to the lesson’s hardening section. If the authorized keys file’s permissions are too open, the server does not read the file and refuses; it also refuses if the public key is not in the file. From the server’s point of view, both are in the “the offered key is not acceptable” branch. What will show the difference between them is not a test, it is what the server writes on that branch — and what it writes depends on the log’s verbosity level.

This is the access layer’s form of the course’s third claim. Kept at the basic level, the log gives three distinct answers and eliminates four candidates; at the verbose level it gives five answers and eliminates seven. The difference is hidden in a three-line output. But the level must have been raised before the fault: detail turned on afterward does not bring back a past attempt. The evidence does not wait.

One more row in the table reinforces this. The four tests reachable from outside produce sixty lines and leave the largest unseparated group at four members. The two tests running on the server produce fifteen lines and leave the largest group at two members. All of the elimination power sits on the server; the problem is that getting onto the server is closed. The answer to this is leaving a second path alongside secure shell: a console that can be logged, or a separate management channel. Without that path, diagnosis stays limited to the four tests reachable from outside, and the four-member group stays exactly as it is.

Candidate Set Sweep

There is no second seed in this course, because there is no randomness. In its place, the candidate list is changed. The table below puts three lists side by side, and what needs reading is not the eliminated column but the remaining column.

When account-locked is removed from the list, the number of candidates the verbose log eliminates drops from seven to six; what remains is two, and the indistinguishable group count drops from two to one. When method-disabled is added to the list, the eliminated count does not change — still seven — but what remains rises from two to three, and the largest indistinguishable group widens from two members to three. Same test, same answer, same elimination count; the diagnosis got worse.

The rule that follows is this: elimination count alone is not a measure. As the candidate list grows, the same test eliminates the same number of candidates and leaves more behind. What needs to be measured is not elimination but remaining ambiguity. This is the second half of the course’s rule, and it will come up again and again in the following lessons as the jump server, chains, and address translation grow the candidate list.

Summary

• Two authentications stand stacked in a secure shell connection: the client authenticates the server with the host key, the server authenticates the user with the public key. The private key never leaves the client.
• Hardening settings change the candidate-cause list: turning off password authentication removes password-related candidates, the authorized keys file’s permissions add a new candidate.
• Across nine candidate causes and seven tests, the eliminated counts are not equal: the verbose server log eliminates 7 candidates in three lines, the verbose client output 5 in forty-six lines, the listening list 1 in twelve lines.
• When the seven tests are used together, two groups cannot be separated; what does the separating is not a tool but the log’s verbosity level, raised before the fault. The basic level eliminates 4, the verbose level 7 candidates.
• When one item is added to the candidate list, the eliminated count stays fixed while what remains rises from 2 to 3; what needs to be measured is not elimination but remaining ambiguity.

Next Step

This lesson’s sharpest result was that the elimination power sits on the server, and getting onto the server is closed. The four tests reachable from outside leave a four-member group exactly as it is. The next lesson takes up the usual solution to this blockage: going to a machine that cannot be reached directly, by way of another machine that can be reached. A jump server and a tunnel split the path in two, and they split the diagnosis in two as well — the fault can now be on either of two machines. The question to measure is how many of this expanded candidate set a single end-to-end attempt eliminates.