File Transfer Protocols

The previous three lessons set the three roads to real-time behavior side by side and measured the round per decision. All three shared one thing: each sat between the two ends of a single application, and each used the same transport envelope. The intermediary was looking at a single connection; what it saw, it either saw whole or not at all.

Some protocols do not work this way. A file transfer can split a single logical job across two separate connections: commands on one, the file’s bytes on the other. This lesson’s question is: if the intermediary sees both but has no field telling it which belongs to which, what does it lose? The loss has nothing to do with hiding; this lesson counts those two costs separately.

• OP41. The setup is M04/K03’s shared setup: the station.example measurement station, and an intermediary in between that does not know the application. The intermediary looks only at what is written in the message.
• OP42. The set measured is 40 transfers and 5 decisions per transfer: is_upload, retryable, is_measurement_file, resumable, completed. Total 200 decisions; the smallest measurable difference in this set is $1/200 = 0{,}0050$.
• OP43. The oracle — the truth itself — is known because we produced the setup ourselves, and it does not change from regime to regime. The regime only changes the fields the intermediary sees.
• OP44. Four regimes are set up: two channels open, data channel wrapped, single channel open, single channel wrapped. The transfers themselves are the same across all four regimes.
• OP45. The use of transfer tools is not repeated here. Remote shell, key transport and synchronization procedure were measured in the Linux Network Administration and Troubleshooting course; the measure here is channel separation, not the tool.

The Control Channel and the Data Channel

The classic form of file transfer defines two channels. The control channel carries commands and reply codes; it stays open for the whole session. The data channel carries only the bytes of the file or the directory listing; it opens for each transfer and closes when the transfer ends. FTP is the specification that makes this split.

control channel — between the client and station.example, open for the whole session

  ->  USER olcum
  <-  331 password required
  ->  PASS ********
  <-  230 session opened
  ->  TYPE I
  <-  200 type binary
  ->  PASV
  <-  227 passive mode (a1,a2,a3,a4,p1,p2)
  ->  SIZE /olcum/kuzey.csv
  <-  213 9216
  ->  RETR /olcum/kuzey.csv
  <-  150 opening data channel
  <-  226 transfer complete

data channel — a separate connection, for the duration of the transfer

  9216 bytes flow. No command, no path, no session marker; the connection closes when it ends.

This dump is not run; it is the taught form. Two points should be read. First, where the data channel will open is negotiated in the control channel’s body: with PORT the client announces where it is listening itself, with PASV the server announces where it has opened its own end. Second, the bytes flowing on the data channel carry no field that says which command they belong to.

Two designs sit at the opposite end. SFTP opens a single connection, wraps that connection in an envelope, and frames both commands and bytes over the same connection; every request carries a request marker and the response comes back with that marker. FTPS keeps the two-channel layout but wraps each channel in its own envelope — and “each” is the key word here.

Resume and Directory Listing

Two of the five measured decisions come from these two details, so they need to be set up first.

Resume. A transfer cut short partway can be completed in two ways: from the start, or from where it left off. Resuming from where it left off is done by the client declaring a starting offset and repeating the command; in the specification the command for this is REST. Whether the server supports resuming is a capability declared on the control channel, and nothing verifies the declaration. This decision has a concrete cost for the intermediary: if resuming is supported, a cut-short transfer only costs the remaining bytes; if not, the whole file flows again. In the measurement every transfer is a single attempt; resume’s effect on byte accounting should also be noted separately, because in a resumed transfer the byte count flowing on the data channel is smaller than the declared size, and an intermediary that looks at the size comparison mistakes a successful transfer for an incomplete one.

Directory listing. The format of the directory listing the LIST command returns on the data channel is not fixed in the specification; it is designed for a human to read. The result is the sharpest form of the course’s rule: the field exists, but it has no meaning. The intermediary sees the bytes, can even split the lines, but has no contract it can rely on for which column is size and which is name. The specification closes this gap later with a listing command meant to be read by machine — MLSD; the way it closes the gap is not a new byte, it is fields whose name and meaning are fixed. In the measurement, LIST accordingly sits in the same class as RETR on the retryable side: neither changes the station’s state.

The Fields in the Intermediary’s Hands

The intermediary reads the command, the path, the size the server declares, and the resume support from the control channel. From the data channel it reads only a byte count. To combine these two observations it has exactly one thing at hand: time proximity. It counts the connection opened right after a command as that command’s connection.

Proximity is not a substitute for a marker. If another session is also connected to the same station, the connection the intermediary counts as adjacent may belong to someone else. The measurement counts this.

The difference between active and passive mode is, from the intermediary’s side, a matter of direction: in active mode the server opens the data connection, in passive mode the client does. The intermediary sees both, but what it sees is a connection setup, not a command. Who opens the connection and which way the bytes flow are two separate things, and the second cannot be derived from the first. The direction of the bytes is read only from the command on the control channel: RETR downloads, STOR uploads. This is the only source of the is_upload decision measured here, and when the command is invisible, that decision goes too — this is why the fourth regime’s zero is zero.

SEED = 20260809
COMMAND_POOL = ["RETR", "RETR", "STOR", "LIST", "RETR", "STOR", "RETR"]
PATH_POOL = ["/olcum/kuzey.csv", "/olcum/yamac.csv", "/ozet.csv",
             "/kayit.log", "/olcum/gunluk.csv"]
DECISIONS = ("is_upload", "retryable", "is_measurement_file",
             "resumable", "completed")


def generator(seed):
    d = seed % 2147483646 + 1

    def r(n):
        nonlocal d
        d = (d * 48271) % 2147483647
        return d % n
    return r


def transfers(count=40, seed=SEED):
    r, out = generator(seed), []
    for i in range(count):
        command, path = COMMAND_POOL[r(7)], PATH_POOL[r(5)]
        declared = 1024 * (1 + r(9))
        cut = r(5) == 0
        resumable = r(3) != 0
        out.append({
            "no": i + 1, "command": command, "path": path,
            "declared": declared,
            "received": declared - 512 if cut else declared,
            "resumable": resumable,                     # truth
            "resume_flag": resumable and r(7) != 0,      # declaration
            "neighbor_session": r(5) == 0,               # adjacent connection belongs to someone else
            "neighbor_bytes": 1024 * (1 + r(9)),
        })
    return out


def oracle(a):
    return {
        "is_upload": a["command"] == "STOR",
        "retryable": a["command"] in ("RETR", "LIST"),
        "is_measurement_file": a["path"].startswith("/olcum/"),
        "resumable": a["resumable"],
        "completed": a["received"] == a["declared"],
    }


def message(a, visible):
    full = {
        "command": a["command"], "path": a["path"], "host": "station.example",
        "resume_flag": a["resume_flag"], "declared": a["declared"],
        "adjacent_bytes": a["neighbor_bytes"] if a["neighbor_session"] else a["received"],
        "channel_bytes": a["received"], "session_marker": a["no"],
    }
    return {k: v for k, v in full.items() if k in visible}


def infer(i):
    k = {}
    if "command" in i:
        k["is_upload"] = i["command"] == "STOR"
        k["retryable"] = i["command"] in ("RETR", "LIST")
    if "path" in i:
        k["is_measurement_file"] = i["path"].startswith("/olcum/")
    if "resume_flag" in i:
        k["resumable"] = i["resume_flag"]
    if "declared" in i:
        if "session_marker" in i:
            k["completed"] = i["channel_bytes"] == i["declared"]
        elif "adjacent_bytes" in i:
            k["completed"] = i["adjacent_bytes"] == i["declared"]
    return k


CONTROL = ("command", "path", "host", "resume_flag", "declared")
VISIBLE = {
    "two channels open": CONTROL + ("adjacent_bytes",),
    "data channel wrapped": CONTROL,
    "single channel open": CONTROL + ("channel_bytes", "session_marker"),
    "single channel wrapped": ("host",),
}


def measure(name):
    correct = wrong = unavailable = extra_round = 0
    breakdown = {}
    for a in transfers():
        truth, given = oracle(a), infer(message(a, VISIBLE[name]))
        missing = False
        for decision in DECISIONS:
            if decision not in given:
                unavailable += 1
                missing = True
            elif given[decision] == truth[decision]:
                correct += 1
            else:
                wrong += 1
                breakdown[decision] = breakdown.get(decision, 0) + 1
        extra_round += 1 if missing else 0
    return correct, wrong, unavailable, extra_round, breakdown


print(f"{'regime':<23s} {'correct':>7s} {'wrong':>6s} {'unavailable':>11s} "
      f"{'round':>5s}  wrong breakdown")
for name in VISIBLE:
    c, w, u, e, dk = measure(name)
    print(f"{name:<23s} {c:7d} {w:6d} {u:11d} {e:5d}  {dk}")

a = transfers()
print()
print(f"transfers {len(a)}, decisions {len(a) * len(DECISIONS)}, "
      f"cut short {sum(x['received'] != x['declared'] for x in a)}, "
      f"left unflagged {sum(x['resumable'] and not x['resume_flag'] for x in a)}, "
      f"neighbor session {sum(x['neighbor_session'] for x in a)}")

regime                  correct  wrong unavailable round  wrong breakdown
two channels open           192      8           0     0  {'completed': 3, 'resumable': 5}
data channel wrapped        155      5          40    40  {'resumable': 5}
single channel open         195      5           0     0  {'resumable': 5}
single channel wrapped        0      0         200    40  {}

transfers 40, decisions 200, cut short 7, left unflagged 5, neighbor session 6

The Cost of Splitting Is Not the Cost of Hiding

The first row and the third row see the same amount of fields: the same command, the same path, the same declared size, the same byte count. Nothing is hidden. Yet the two-channel regime takes 192 correct decisions while the single-channel regime takes 195. The 3 decisions between them come from splitting alone, and only splitting: on a single channel the bytes are framed and carry a session marker, so matching is correct by construction; on two channels, the intermediary has mistaken a neighboring session’s connection for this command’s connection. 3/200 = 0.015; three times the set’s resolution, so a measured difference.

There is a subtlety in reading the number. A neighboring session steps in on 6 transfers, but the wrong decision only comes out of 3 of them. In the remaining three, the transfer itself is already cut short: the intermediary says “not complete,” the oracle also says “not complete,” and the decision comes out correct. The intermediary gave the right answer for the wrong reason. The measurement counts this as correct, because what is measured is the decision; but from a design standpoint this is not a guarantee, it is two errors covering for each other.

The second row says something entirely different. When the data channel is wrapped, the intermediary cannot see the byte count at all, never takes the completed decision, and in forty of the forty transfers it owes the server one more round. A misunderstanding should be closed here: a wrapped connection still carries a byte count on the wire, but that number is not the file’s size — the envelope’s record headers and padding are inside that total too. The intermediary’s number and the declared size on the control channel are not even in the same unit; this is why the decision does not come out wrong, it does not come out at all. In return, 3 wrong decisions disappear: the remaining 5 wrong ones all come from another source. This is this lesson’s form of the course’s third claim — fewer decisions, but fewer wrong ones too. A wrapper is not a defect, it is a choice, and the choice’s name is: the round debt is paid, the wrong match is not bought.

The source of the remaining 5 wrong decisions is neither the split nor the wrapper. In five transfers the server genuinely supports resuming but has not declared it on the control channel. The intermediary looks at the declaration, says “not resumable,” and is wrong. This is the course’s fourth reading, repeated in every lesson: nothing verifies what is written in the message. If a field is missing, the intermediary does not know; if it is wrong, it knows wrong. This same five is the same five in three of the four regimes — splitting and hiding neither raise nor lower it.

Why the Insecure Option Is Insecure

The base specification carries both the control channel and the data channel with no envelope at all. The result is: every device standing along the path can read the credential passed at session opening, the requested file paths, the declared sizes, and the file’s own bytes. This is not a defect, it is the assumption the specification was written under — the path itself was assumed trustworthy. When the assumption does not hold, the protocol has no field to close it.

Splitting adds a second point here: the two channels are protected separately. In a setup where the control channel is wrapped and the data channel is left open, the client assumes its session is protected, while the file’s bytes are unprotected. The measurement’s second row is the mirror image of this: there the data channel is wrapped and the control channel is open; the intermediary keeps reading the path and the size. In a two-channel design, the question “is it protected” has no single answer — it has one answer per channel.

The third point is structural. Because the data channel’s address is negotiated in the control channel’s body, an intermediary translating the address has to read and rewrite that body. When the control channel is wrapped, it cannot do this; the transfer either cannot be set up, or the configuration is forced to keep a whole port range permanently open. Hiding and the middle device’s ability to do its job stand in direct conflict here.

Narrowing. All three of the surfaces counted point the same way: reduce the channel count to one. When a single connection is wrapped in an envelope, the credential, the path, the size and the file’s bytes are all protected by one decision; framing settles the matching by construction, so the three wrong decisions never arise; and because there is no address negotiation in the body, the middle device does not need to rewrite the body. SFTP goes this way. If the two-channel layout has to be kept, the narrowing is asking for the protection of both channels separately and explicitly; one being wrapped says nothing about the other. The measurement’s last row also writes down the price: in the single-channel-wrapped regime the intermediary loses all 200 decisions. Its caching, retrying and size checking are absent in this regime; that work returns to the endpoints.

Summary

• Classic file transfer splits one job across two connections: the control channel carries commands and reply codes, the data channel carries only the bytes; nothing on the bytes says which command they belong to.
• On the set of 40 transfers and 200 decisions, the two-channel open regime takes 192 correct decisions and the single-channel open regime takes 195; the 3 decisions between them come only from splitting, not from hiding.
• A neighboring session steps in on 6 transfers but the wrong decision only arises in 3; in the remaining three the transfer is already cut short, so the intermediary gives the right answer for the wrong reason.
• When the data channel is wrapped, the completed decision cannot be taken 40 times and 40 round debts arise; in return, 3 wrong decisions disappear — fewer decisions, but fewer wrong ones.
• The 5 wrong decisions are the same across all three regimes and come from the server not declaring its resume support; neither splitting nor hiding repairs a missing declaration.
• In the unwrapped base specification the credential, the path, the size and the file’s bytes can all be read along the path; because the two channels are protected separately, the question “is it protected” has one answer per channel, and the narrowing is to reduce the channel count to one.

Next Step

What was split in this lesson was two channels: one carried what was requested, the other what arrived. When the intermediary could not match them, it got a decision wrong. In the next lesson what is split will be two names. The fields that say where a piece of mail is to be delivered and the fields that show the user who it came from sit in the same message, do not verify each other, and delivery is routed by looking at the envelope. The name the intermediary reads for routing and the name the human sees on screen come from two separate fields; the next lesson measures why this split exists and which control closes it.