Mail Protocols

The previous lesson measured a protocol that splits one job across two channels: commands in one, bytes in the other, and the intermediary got a decision wrong when it could not match the two. The loss came from the split, not from hiding.

Mail goes one step further. Here what splits is not the channel but the name. The fields that say where a piece of mail is to be delivered and the fields that show the user who it came from sit inside the same message, in two separate layers. The layer that routes delivery is the mail envelope, and the intermediary looks at it. The name the user sees on screen comes from the header and plays no part in delivery. Neither verifies the other. This lesson’s question is why this split exists and which control closes it.

• OP46. The setup is M04/K03’s shared setup: the station.example measurement station produces alert mail, the domains example.test and list.example stand for the recipient side and the list side, and an intermediary that does not know the application sits in between.
• OP47. The mail envelope and the transport envelope are two separate concepts. The transport envelope is the layer built in the Secure Transport topic that blinds the intermediary. The mail envelope, in contrast, is the protocol’s own fields; it hides nothing, it only routes delivery. What is measured in this lesson is the mail envelope.
• OP48. The set measured is 40 messages and 5 decisions per message: delivery_path_exists, sender_authorized, names_aligned, has_bcc_recipient, displayed_name_verified. Total 200 decisions; the smallest measurable difference is $1/200 = 0{,}0050$.
• OP49. Messages are of four kinds: direct, proxied (sent by another domain on the station’s behalf), list, and forwarded. All four are legitimate; there is no forged message anywhere in the setup. The measurement’s purpose is not to count forgeries, it is to count what legitimate variety does to controls.
• OP50. Publishing a domain policy is the domain’s own choice: station.example and list.example publish one, example.test does not. The oracle — the truth itself — is known because we produced the setup ourselves and does not change from regime to regime.

Submission and Retrieval

Mail is carried by two separate protocol families, and they face opposite directions.

Submission is the road that pushes the message from the client to the recipient’s mail store. SMTP is this road’s specification; the same specification is used both for the client’s first submission and for the transfer between relays afterward. Retrieval, in turn, is the road the user pulls a message from once it has reached the store; IMAP and POP3 are this direction’s specifications. IMAP keeps the message on the server and the user reads the same view from many places along with the folder structure; POP3 is built on downloading the message and dropping it from the store.

The consequence for this lesson is: the intermediary only stands on the submission road. On the retrieval road there is no decision-making point between the user and their own store. A decision not taken during submission cannot be taken back during retrieval — the message has already been delivered. The round’s debt is written into a mailbox here.

The two retrieval specifications also diverge in round behavior. In POP3 the state lives in the client: message numbers inside a session are specific to that session, and once messages are downloaded and dropped from the store, a second client cannot see the same mailbox. In IMAP the state lives on the server; every message carries a persistent identifier that does not change from session to session, along with flags such as read, replied, deleted, so multiple clients share the same view. The price is this: every query on shared state is a round trip. The previous three lessons’ polling question comes up on the retrieval side too, in exactly the same shape, and IMAP’s answer belongs to the same family — the IDLE command holds the connection open, the server reports the change itself, and the client wastes no round. Short polling’s round cost per decision holds here too.

The Mail Envelope and the Header

The two layers of the submission conversation sit side by side below.

submission conversation — between the client and the delivering host

  ->  EHLO station.example
  <-  250 hello
  ->  MAIL FROM:<[email protected]>       mail envelope: return path
  <-  250 sender accepted
  ->  RCPT TO:<[email protected]>             mail envelope: delivery address
  <-  250 recipient accepted
  ->  RCPT TO:<[email protected]>             mail envelope: second recipient
  <-  250 recipient accepted
  ->  DATA
  <-  354 send the message, end with a single dot

      From: Kuzey Yamac Istasyonu <[email protected]>   header
      To: Nobet <[email protected]>                         header
      Subject: threshold exceeded                          header
      Date: ...

      The measurement exceeded the threshold.
      .
  <-  250 message queued

This dump is not run; it is the taught form. Four points should be read.

First, delivery is done only by looking at the RCPT TO lines. The To: header plays no part in delivery; a delivery that does not read it still works correctly. Second, the message above has two envelope recipients but the header names only one — [email protected] never enters the header at all. A blind carbon copy is not a header field, it is an envelope recipient; there is no corresponding line for it in the header, and there should not be. Third, MAIL FROM and From: are two separate addresses. The first is where an undeliverable message will bounce back to, the second is the name to be shown to the user. In a list message, the return path belongs to the list and the displayed name belongs to the original author; the two being different is not a protocol defect, it is the design.

A fourth point shows up for the first time in this course: the intermediary here does not only read, it writes too. Every host that relays the message adds a Received: line to the top of the header, and the accumulating lines record, in reverse order, the path the message took. This is the only place in the course where the intermediary puts its own declaration into the message. The record has a limit worth noting for this lesson: Received: lines are a self-declaration by the hosts that carried the message, and nothing verifies the lines below the top one. A trace existing does not mean the trace is correct — the course’s fourth reading holds here exactly the same way.

The Measurement

The intermediary is measured at three levels of visibility: seeing only the mail envelope, also reading the headers, and also having received the key that signs the header.

SEED = 20260809
TYPES = ["direct", "direct", "direct", "proxied",
         "list", "forwarded", "direct"]
ENVELOPE_DOMAIN = {"direct": "station.example", "proxied": "example.test",
                    "list": "list.example", "forwarded": "example.test"}
POLICY = {"station.example": True, "list.example": True, "example.test": False}
DECISIONS = ("delivery_path_exists", "sender_authorized", "names_aligned",
             "has_bcc_recipient", "displayed_name_verified")


def generator(seed):
    d = seed % 2147483646 + 1

    def r(n):
        nonlocal d
        d = (d * 48271) % 2147483647
        return d % n
    return r


def messages(count=40, seed=SEED):
    r, out = generator(seed), []
    for i in range(count):
        kind = TYPES[r(7)]
        envelope_recipients = 1 + r(5)
        out.append({
            "no": i + 1, "type": kind,
            "envelope_domain": ENVELOPE_DOMAIN[kind],
            "header_domain": "station.example",
            "envelope_recipients": envelope_recipients,
            "header_recipients": envelope_recipients - 1 if r(5) == 0 else envelope_recipients,
            "forwarded": kind in ("list", "forwarded"),
            "source_authorized": r(9) != 0,     # truth: the exit host is actually authorized
            "signature_present": r(7) != 0,
            "altered": r(9) == 0,                # truth: the message was altered in transit
            "path_exists": r(11) != 0,
        })
    return out


def oracle(m):
    return {
        "delivery_path_exists": m["path_exists"],
        "sender_authorized": m["source_authorized"],
        "names_aligned": m["envelope_domain"] == m["header_domain"],
        "has_bcc_recipient": m["envelope_recipients"] > m["header_recipients"],
        "displayed_name_verified": m["signature_present"] and not m["altered"],
    }


def message(m, visible):
    full = {
        "envelope_domain": m["envelope_domain"], "envelope_recipients": m["envelope_recipients"],
        "path_exists": m["path_exists"],
        "deliverer_authorized": m["source_authorized"] and not m["forwarded"],
        "header_domain": m["header_domain"], "header_recipients": m["header_recipients"],
        "signature_present": m["signature_present"],
        "signature_intact": m["signature_present"] and not m["altered"],
    }
    return {k: v for k, v in full.items() if k in visible}


def infer(i):
    k = {}
    if "path_exists" in i:
        k["delivery_path_exists"] = i["path_exists"]
    if "envelope_domain" in i and POLICY[i["envelope_domain"]]:
        k["sender_authorized"] = i["deliverer_authorized"]
    if "envelope_domain" in i and "header_domain" in i:
        k["names_aligned"] = i["envelope_domain"] == i["header_domain"]
    if "envelope_recipients" in i and "header_recipients" in i:
        k["has_bcc_recipient"] = i["envelope_recipients"] > i["header_recipients"]
    if "signature_intact" in i:
        k["displayed_name_verified"] = i["signature_intact"]
    return k


ENVELOPE = ("envelope_domain", "envelope_recipients", "path_exists", "deliverer_authorized")
HEADER = ("header_domain", "header_recipients", "signature_present")
VISIBLE = {
    "envelope": ENVELOPE,
    "envelope+header": ENVELOPE + HEADER,
    "envelope+header+key": ENVELOPE + HEADER + ("signature_intact",),
}


def measure(name):
    correct = wrong = unavailable = extra_round = 0
    breakdown = {}
    for m in messages():
        truth, given = oracle(m), infer(message(m, VISIBLE[name]))
        missing = False
        for decision in DECISIONS:
            if decision not in given:
                unavailable += 1
                missing = True
            elif given[decision] == truth[decision]:
                correct += 1
            else:
                wrong += 1
                breakdown[decision] = breakdown.get(decision, 0) + 1
        extra_round += 1 if missing else 0
    return correct, wrong, unavailable, extra_round, breakdown


print(f"{'visibility':<22s} {'correct':>7s} {'wrong':>6s} {'unavailable':>11s} "
      f"{'round':>5s}  wrong breakdown")
for name in VISIBLE:
    c, w, u, e, dk = measure(name)
    print(f"{name:<22s} {c:7d} {w:6d} {u:11d} {e:5d}  {dk}")

m = messages()
misaligned = [x for x in m if x["envelope_domain"] != x["header_domain"]]
print()
print(f"messages {len(m)}, decisions {len(m) * len(DECISIONS)}, misaligned {len(misaligned)}, "
      f"all misaligned are legitimate "
      f"{all(x['type'] in ('proxied', 'list', 'forwarded') for x in misaligned)}")
print(f"forwarded {sum(x['forwarded'] for x in m)}, "
      f"from a domain with a published policy {sum(POLICY[x['envelope_domain']] for x in m)}, "
      f"carrying a bcc recipient {sum(x['envelope_recipients'] > x['header_recipients'] for x in m)}")
intact = sum(x["signature_present"] and not x["altered"] for x in misaligned)
print(f"misaligned with intact signature {intact}, "
      f"unsigned {sum(not x['signature_present'] for x in misaligned)}, "
      f"altered in transit {sum(x['signature_present'] and x['altered'] for x in misaligned)}")

visibility             correct  wrong unavailable round  wrong breakdown
envelope                    60      7         133    40  {'sender_authorized': 7}
envelope+header            140      7          53    40  {'sender_authorized': 7}
envelope+header+key        180      7          13    13  {'sender_authorized': 7}

messages 40, decisions 200, misaligned 20, all misaligned are legitimate True
forwarded 14, from a domain with a published policy 27, carrying a bcc recipient 8
misaligned with intact signature 17, unsigned 2, altered in transit 1

Reading the Numbers

The envelope governs delivery and says nothing about the header. An intermediary that sees only the mail envelope gets 60 of the 200 decisions; it never gets the remaining 133 at all, and in forty of the forty messages it owes a round. The decisions it gets are the delivery-related ones — does a path exist, is the sender authorized. What the user will see on screen can never be derived from this visibility. This is the cleanest proof of the split: the information delivery needs and the information display needs are separate sets.

Reading the header buys 80 decisions and adds not a single round. Headers are already flowing on the same connection, right after DATA; when the intermediary reads them, correct decisions climb from 60 to 140. This is the reverse face of the course’s second claim: adding a field does not buy a decision, but reading an already-carried field can. What buys the gain is not the byte itself, it is being a field whose name and meaning are fixed.

Misalignment alone says nothing. In 20 of 40 messages the envelope domain and the header domain differ, and in the setup all twenty are legitimate: proxied submission, list mail, forwarded mail. A rule that filters purely on misalignment eliminates half of this set. The intermediary gives the names_aligned decision correctly every single time — but the question it is correctly answering is not the question anyone was asking.

Path-based checking breaks with forwarding. Seven decisions come out wrong, and all seven come from the same source: path-based checking looks at the host that delivered the message. In list mail the delivering host is the list’s own host, even though the message genuinely came from an authorized source. The intermediary says “not authorized” and is wrong. For the 13 messages coming from a domain that does not publish a policy, the decision is never taken at all — the price this lesson charges for a missing declaration is not a wrong answer, it is a gap.

The signature round is paid once. Verifying the header-covering signature costs one key-fetching round; after that round the intermediary reaches 180 correct decisions, and the number of round-indebted messages drops from 40 to 13. The key is fetched not per message but per domain: since this set has a single header domain, the round is paid once and covers all forty messages. The remaining 13 come from domains that have not published a policy; their debt is closed not by the signature, but by publication.

Why the Split Exists, Which Control Closes It

The envelope and the header being separate is not a slackness added on afterward — it is a split delivery needs in order to work. Where an undeliverable message bounces back to has to be independent of the name that will be shown; otherwise a list message’s bounce would fall into the original author’s mailbox. A list or a forward has to be able to route the message onto another path without changing the recipient set. A blind carbon copy has to be deliverable without being written into the header. All three require the envelope to be separate from the header.

The price of the split is that nothing verifies the name written in the header. The measurement’s 20 misaligned messages are its legitimate face; the same structure means the trust the displayed name carries does not come from the protocol itself. The control that closes this has three parts, and each answers a separate question:

• Sender policy (SPF) declares which hosts the envelope domain’s mail is allowed to leave from. It is path-based, so it breaks with forwarding — all 7 of the measurement’s wrong decisions come from here. If unpublished, no decision is taken: 13 messages.
• Header-covering signature (DKIM) binds the displayed name to the domain that signs it. It survives forwarding, because it looks at content, not path; in the measurement, 17 of the 20 misaligned messages carry an intact signature. It breaks if the content is altered in transit: 1 message. Its price is one key-fetching round.
• Alignment and reporting policy (DMARC) aligns the result of the first two with the header domain and declares what the domain owner wants done. It produces no proof on its own; it binds the output of two controls into a single decision.

Narrowing. Every one of the surfaces counted closes in the same direction: move the decision from path to content, and align the result with the header domain. Path-based checking alone flags forwarding as false; content-based signing alone may not align with the displayed name; alignment policy alone produces nothing to verify. Together, the three raise not the count of decisions the intermediary takes without a round trip but their correctness, and in exchange they ask for one round per domain. This is exactly this lesson’s measure too: the intermediary already saw the misalignment; what was missing was not seeing it, it was a declaration to bind what it saw.

Summary

• Mail is carried by two families facing opposite directions: submission pushes the message into the store (SMTP), retrieval pulls it out of the store (IMAP, POP3); the intermediary only stands on the submission road, and a decision not taken there cannot be taken back afterward.
• The mail envelope governs delivery (MAIL FROM, RCPT TO), the header carries the name shown to the user (From:, To:); the two are separate fields, they do not verify each other, and a blind carbon copy exists only in the envelope.
• An intermediary that sees only the envelope gets 60 of 200 decisions and owes 40 rounds; also reading the headers raises correct decisions to 140 without adding a single round.
• In 20 of 40 messages the names are misaligned and all twenty are legitimate; a rule that looks only at misalignment eliminates half the set — the intermediary gives the right answer to a question no one asked.
• Path-based sender policy breaks with forwarding and flags 7 decisions wrong, and produces no decision for the 13 unpublished messages; the header-covering signature binds 17 of the 20 misaligned messages and its price is a single round per domain.

Next Step

What these two lessons measured was what the intermediary could read from a message. Reading has one more precondition: the timestamps inside the message have to show the same clock. The next lesson measures the protocol that aligns clocks, and opens an assumption this course has not chased down for a single round yet — the assumption that the outbound delay and the return delay are equal. If they are not equal, clock offset follows directly from that gap, and the protocol has no field in its hands to see the difference. The same lesson closes the course by comparing two management models — polling devices against devices that report — on round and decision balance.