--- title: 'Application Layer Protocols' source: 'https://academia.sh/en/courses/application-protocols' language: en updated: '2026-08-17T04:39:03+00:00' license: 'CC BY-SA 4.0' lessons: 16 outcomes: - 'Explains the design differences between HTTP versions in terms of performance' - 'Reads the steps of the TLS handshake and the certificate chain' - 'Selects the appropriate protocol for real-time communication' --- # Application Layer Protocols The method, status, header and caching rules of the HTTP family; the key, certificate and handshake chain of secure transport; real-time, file, mail and management protocols — each measured by the decision an intermediary can draw from the message. ## The HTTP Family The commitment a method carries, the classes behind status codes, content negotiation through headers, a cookie's scope rules, the framing difference across versions, and cache validation. ### HTTP Methods and What They Mean Source: https://academia.sh/en/courses/application-protocols/http-methods-and-what-they-mean A method is a declaration: the intermediary in between draws the repeatable and storable decisions from it, and nothing verifies the declaration. The Network Models and Protocols course closed out the transport layer with a single promise left in hand: TCP delivers an ordered, lossless, non-duplicated byte stream. A gap sits inside that promise. The transport layer carries **bytes**; it does not know what a byte means. The same forty bytes could be a measurement record or a delete command; TCP does not tell the two apart, nor take on that job. Meaning is built by the **application layer**, and the thing that builds it is nothing other than what is written in the message: a method name, a path, a header field. This is exactly what gets counted throughout this course — how many decisions what is written in the message lets a third party reading it make. This lesson first sets up that third party and the measure, then measures the first field: the method. #### The Intermediary: The Layer That Does Not Know the Application The North Slope measurement station serves its readings to a client. Between the two stands an **intermediary**: a layer that caches, routes, and retries when needed. A **proxy** is one kind of intermediary; the general name is intermediary. The intermediary does not know the application: not what was measured, what a record means, or which session belongs to whom. All it knows is what is written in the message. The whole subject of this course fits in one sentence: **whatever is written in the message, the intermediary knows; whatever is not written, it does not know; and whatever is written wrong, it knows wrong.** The request an intermediary sees looks like this. The dump below is for teaching purposes and is not run: ```text POST /register HTTP/1.1 Host: station.example Content-Type: application/json Content-Length: 74 ``` On the response side there are two more markers the intermediary reads. The response's **first line** is next lesson's subject; only header fields are read here: ```text Cache-Control: private ETag: "s-0417" ``` These four lines correspond to the following fields in the model: | Where in the dump | Field in the model | What the intermediary derives | |---|---|---| | First word of the request line | `method` | `repeatable`, `storable` | | Second word of the request line | `path` | the resource's identity | | `Host` | `host` | `redirectable` | | `Cache-Control: private` | `private_marker` | `shareable`, `storable` | | `ETag` | `validator` | `fresh` | #### A Method Is a Declaration A method declares two separate properties. If a method is **safe**, sending the request is not meant to change state on the server; if **idempotent**, sending it once or five times in a row leaves the same result. | Method | Safe | Idempotent | What the intermediary derives | |---|---|---|---| | GET | yes | yes | response storable, request repeatable | | HEAD | yes | yes | response storable, request repeatable | | PUT | no | yes | response not storable, request repeatable | | DELETE | no | yes | response not storable, request repeatable | | POST | no | no | neither | Safety implies idempotence: a request that does not change state does not change it on a repeat either. The reverse does not hold; PUT changes state, but repeating it with the same body leaves the same state. The resource design topic of the Web API Design course treats this distinction as **the designer's commitment** and says which method should be chosen. Here the choice is already made; what is measured is what that choice **tells the intermediary**. The idempotence key is established in the same course too, as an application contract; this lesson measures the protocol's own declaration, not what the application adds on top. #### The Measure: Decisions Taken Without an Extra Round Trip A protocol's number is not the fields it carries but **how many decisions it lets the receiving side make without an extra round trip.** A decision that cannot be made is a round-trip debt: the intermediary must ask what it cannot know, and asking costs a round trip. The model produces forty exchanges, each asking the intermediary to make five decisions. | Decision | Question | |---|---| | `storable` | May this response be cached | | `shareable` | May the cached copy be given to another client | | `fresh` | Is the copy still valid | | `repeatable` | May the request be resent if it goes unanswered | | `redirectable` | Can the message be routed to its destination | Forty exchanges and five decisions make a set of two hundred decisions. This set also sets the measurement's **resolution**: the smallest difference measurable here is $1/200 = 0.0050$, none smaller is claimed. The truth itself is represented by the **oracle**: because we produced the setup, whether each exchange is actually personalized and whether the copy actually went stale is known. The intermediary's decision is counted against this truth as correct, wrong, or unavailable. Assumptions of the measurement: - **HA1** — Forty exchanges are generated from a single seed; method and path are drawn from fixed lists, and the measurement gives the same set every run. - **HA2** — The oracle's truth is known because we produced the setup: whether the response is personalized and whether the copy went stale are set at generation time, not read from the message. - **HA3** — A single regime is measured in this lesson: the message is **open** and all five fields are visible to the intermediary. Regimes where a field is hidden are the subject of later lessons. - **HA4** — The intermediary does not know the application and builds every decision only from a field present in the message. If a field is not in the message, the decision is never made — not a wrong decision, an **unavailable** one. - **HA5** — The private flag is the server's own declaration, and the server sometimes skips setting it on a personalized response; the validator tag also has a resolution and sometimes misses a change. **Nothing verifies what is written in the message.** - **HA6** — The byte count is computed from field name, value and separator length; the measurement is not a network measurement, it is a count of the message's content. #### The Core The setup, the oracle and the intermediary are built in the core below. The `OPEN` tuple gives the fields the intermediary can read: in this lesson the message is open, and all five fields are visible. ```python SEED = 20260809 SAFE = {"GET", "HEAD"} IDEMPOTENT = {"GET", "HEAD", "PUT", "DELETE"} METHOD_POOL = ["GET", "GET", "GET", "HEAD", "POST", "PUT", "DELETE"] PATH_POOL = ["/measurement/north", "/measurement/slope", "/summary", "/register", "/session"] DECISIONS = ("storable", "shareable", "fresh", "repeatable", "redirectable") OPEN = ("method", "path", "host", "private_marker", "validator") def generator(seed: int): d = seed % 2147483646 + 1 def r(n: int) -> int: nonlocal d d = (d * 48271) % 2147483647 return d % n return r def exchanges(count: int = 40, seed: int = SEED) -> list[dict]: r, out = generator(seed), [] for i in range(count): method, path = METHOD_POOL[r(7)], PATH_POOL[r(5)] private = path == "/session" or r(5) == 0 stale = r(3) == 0 out.append({ "no": i + 1, "method": method, "path": path, "private": private, # truth: is the response personalized "stale": stale, # truth: has the copy gone stale "private_marker": private and r(4) != 0, # declaration: did the server set the flag "tag_missed": stale and r(6) == 0, # validator tag missed the change }) return out def oracle(a: dict) -> dict: return { "storable": a["method"] in SAFE and not a["private"], "shareable": not a["private"], "fresh": not a["stale"], "repeatable": a["method"] in IDEMPOTENT, "redirectable": True, } def message(a: dict, visible: tuple) -> dict: full = {"method": a["method"], "path": a["path"], "host": "station.example", "private_marker": a["private_marker"], "validator": a["stale"] and not a["tag_missed"]} return {k: v for k, v in full.items() if k in visible} def byte_count(i: dict) -> int: return sum(len(k) + len(str(v)) + 4 for k, v in i.items()) def infer(i: dict) -> dict: k = {} if "method" in i: k["repeatable"] = i["method"] in IDEMPOTENT if "private_marker" in i: k["shareable"] = not i["private_marker"] if "method" in i: k["storable"] = i["method"] in SAFE and not i["private_marker"] if "validator" in i: k["fresh"] = not i["validator"] if "host" in i: k["redirectable"] = True return k ``` The structure of the `infer` function deserves attention: every decision sits behind an `in` check. If a field is not in the message, the decision never enters the dictionary — the intermediary does not make a wrong decision, it **cannot make one.** This distinction is what gets counted for the rest of the course. #### Measuring the Open Regime The measurement runs right after the core above. In each exchange the oracle's truth is compared against the intermediary's decision; if a decision could not be made, that exchange owes one round trip. ```python batch = exchanges() counter = {k: [0, 0, 0] for k in DECISIONS} # correct, wrong, unavailable extra_round = total_bytes = 0 for x in batch: truth, given = oracle(x), infer(message(x, OPEN)) missing = False for decision in DECISIONS: if decision not in given: counter[decision][2] += 1 missing = True elif given[decision] == truth[decision]: counter[decision][0] += 1 else: counter[decision][1] += 1 extra_round += 1 if missing else 0 total_bytes += byte_count(message(x, OPEN)) print(f"exchanges {len(batch)}, decisions {len(batch) * len(DECISIONS)}, " f"declaring idempotent {sum(x['method'] in IDEMPOTENT for x in batch)}, " f"not declaring idempotent {sum(x['method'] not in IDEMPOTENT for x in batch)}, " f"declaring safe {sum(x['method'] in SAFE for x in batch)}") print(f"personalized {sum(x['private'] for x in batch)}, " f"left unflagged {sum(x['private'] and not x['private_marker'] for x in batch)}, " f"stale {sum(x['stale'] for x in batch)}") print(f"open regime: correct {sum(v[0] for v in counter.values())}, " f"wrong {sum(v[1] for v in counter.values())}, " f"unavailable {sum(v[2] for v in counter.values())}, " f"extra round {extra_round}, bytes {total_bytes}") print() print(f"{'decision':<18s} {'correct':>7s} {'wrong':>7s} {'unavailable':>12s}") for decision, (c, w, u) in counter.items(): print(f"{decision:<18s} {c:7d} {w:7d} {u:12d}") ``` ``` exchanges 40, decisions 200, declaring idempotent 31, not declaring idempotent 9, declaring safe 18 personalized 17, left unflagged 2, stale 12 open regime: correct 195, wrong 5, unavailable 0, extra round 0, bytes 3861 decision correct wrong unavailable storable 40 0 0 shareable 38 2 0 fresh 37 3 0 repeatable 40 0 0 redirectable 40 0 0 ``` The first line to read is **unavailable 0** and **extra round 0**. While the message is open, the intermediary gets two hundred of two hundred decisions and never goes back to the server even once. That is what the three thousand eight hundred and sixty-one bytes buy. The second line is the method's share. **Thirty-one** of the forty exchanges declare an idempotent method, **eighteen** declare a safe one. `repeatable` is correct in all forty decisions, because the oracle and the intermediary both look at the same set: the method name is written in the message and depends on nothing else. In a request left unanswered, this single word alone decides whether the intermediary resends it. `storable` is also correct in all forty, but not for the same reason. This decision depends on two fields at once: the method must be safe **and** the response must not be personalized. **Seventeen** of the forty exchanges are actually personalized. The intermediary cannot infer this from the method; it looks at the flag the server set. #### The Cost of Retrying `repeatable` coming out correct in all forty does not mean it is free; in the case it covers, it is expensive. Say a request was sent and no response came back. All the intermediary knows is that the response did not arrive. It cannot tell whether the request never reached the server, or reached it, was processed, and the response was lost on the way back. The transport layer gives no such distinction: a dropped connection looks the same in both cases. If the method is idempotent, no distinction is needed. Whether the request arrived or not, sending it a second time leaves the same state, so the intermediary resends it without asking anyone. This is the case in **thirty-one** of the forty exchanges, and the decision is made without an extra round trip. The remaining **nine** exchanges use POST. The intermediary cannot resend here, because a second send could double the result of the first. It can only push the decision to the endpoints: report the state to the client, or ask the server. Both are a round trip. A single word zeroes out the round trip in thirty-one exchanges and charges it in nine. This is the measure's first concrete payoff. The same field sits in the same place; what it carries either closes a decision inside the message or converts it into a round-trip debt. A protocol's number is how these two outcomes distribute across forty exchanges. #### Why a Missing Flag Produces Two Wrong Decisions, Not Four The server did not set the flag on **two** of the seventeen personalized responses. In these two exchanges the intermediary trusts the only declaration it has and counts the response as shareable. But the wrong-decision count is two, not four; the reason is in the dump below, which runs after the core and measurement blocks: ```python print(f"{'left unflagged':<21s} {'shareable':>21s} {'storable':>21s}") print(f"{'no':>3s} {'method':<7s} {'path':<12s} {'given':>10s} {'truth':>10s} " f"{'given':>10s} {'truth':>10s}") for x in batch: if x["private"] and not x["private_marker"]: g, v = oracle(x), infer(message(x, OPEN)) print(f"{x['no']:3d} {x['method']:<7s} {x['path']:<12s} " f"{str(v['shareable']):>10s} {str(g['shareable']):>10s} " f"{str(v['storable']):>10s} {str(g['storable']):>10s}") ``` ``` left unflagged shareable storable no method path given truth given truth 1 POST /register True False False False 34 POST /register True False False False ``` Both exchanges use **POST**, which is not safe; the intermediary would not have cached its response with or without the flag, so `storable` comes out **correct** in both — not for the correct reason, but correct. `shareable`, on the other hand, depends only on the flag and comes out wrong in both. The rule that follows is as important as the measure itself: **a missing declaration only costs something where a different decision could otherwise have been made.** If these same two exchanges had used GET, `storable` would also have gone wrong, making the total seven wrong decisions, not five. Decisions at the intersection of two fields are more resilient than decisions tied to one field, because the second field can cover for the first. The remaining three wrong decisions come from the `fresh` decision. Twelve of the forty copies have actually gone stale; in **three** the validator tag failed to catch the change, and the intermediary called a stale copy fresh. The validator's resolution is a later lesson's subject; here only its contribution to the wrong count is counted: two plus three, **five**. #### Nothing Verifies the Declaration Two of the five wrong decisions come from a flag the server did not set, three from a tag's resolution. What they share is this: **neither is the intermediary's fault.** The intermediary read what was written and processed it correctly; the wrong decision comes from what is written parting ways with the truth. There is no mechanism in the protocol that closes this gap. The intermediary cannot test whether a method is actually safe, or see that a request declaring POST changed nothing on the server, or that a request declaring GET deleted a record. All it has is the declaration, and **the intermediary believes the declaration.** There is also a narrowing side to this: when the server does not put a field in the message, the intermediary **cannot** make the decision tied to that field; when it does, the intermediary uses it without question. This choice between the two ends is what gets measured one field at a time in the lessons that follow — what the intermediary gains if a field stays visible, and what it loses if the field disappears. #### Summary - The transport layer carries bytes and does not know their meaning; meaning is built by the application layer, and the only thing that builds that meaning is what is written in the message. - A protocol's measure is not the number of fields it carries but the number of decisions it lets be made without an extra round trip; an unavailable decision is a round-trip debt. - A method declares two things: safety and idempotence. From these the intermediary derives the `repeatable` and `storable` decisions. - On the set of forty exchanges and five decisions, the open regime gives 195 correct, 5 wrong, 0 unavailable decisions and 0 extra round trips; the measurement reads 3861 bytes. - Two of the five wrong decisions come from the two responses left unflagged, and it is two, not four, because those two exchanges' method was already not safe; the remaining three come from changes the validator tag missed. - There is no mechanism that verifies what is written in the message; the intermediary believes the declaration, and the wrong decisions come from that trust. #### Next Step Every field measured in this lesson sat on the request side: method, path, host. The response's headers were looked at, but its **first line** was deliberately left out. Yet the first decision the intermediary makes about a response sits on that line, and what it carries is a three-digit number. The next lesson measures that number: can the intermediary, without opening the body, decide whether to retry or cache just by reading the first digit — and when an intermediary that does open the body reaches the same decision, how many extra bytes does it read to get there. ### Status Codes Source: https://academia.sh/en/courses/application-protocols/status-codes The status line's first digit is an instruction: the intermediary draws its caching, routing and retry decisions from it without ever opening the body. The previous lesson measured the request's fields: method, path, host. The response's header fields were looked at, but its **first line** was deliberately left out. The first decision the intermediary makes about a response sits on that line, and what the line carries is a three-digit number. This lesson's question: how far can the intermediary get, without ever opening the response's body, just by looking at that number's **first digit**. Does an intermediary that also opens the body reach the same decisions — and if so, what does the extra byte it reads buy? #### The Status Line A response opens with a status line. The dump below is for teaching purposes and is not run: ```text HTTP/1.1 200 OK Content-Type: application/json Content-Length: 128 HTTP/1.1 301 Moved Permanently Location: /measurement/north/current HTTP/1.1 503 Service Unavailable Retry-After: 30 ``` The line has three parts: the protocol name and version, the three-digit **status code**, and a short phrase written for a human to read. The only part useful to the intermediary is the number in the middle; the text on the right is informational and does not enter any decision. Even the number itself is not needed whole. The first digit gives the code's **status class**, and that is the actual thing the intermediary reads. #### The Class Itself Is an Instruction | Class | Meaning | What it tells the intermediary | |---|---|---| | 1xx | Informational | Interim response; forward it, do not cache it | | 2xx | Success | The response is fit to be cached | | 3xx | Redirection | The resource's address changed | | 4xx | Client error | The same request gives the same result; retrying is pointless | | 5xx | Server error | If the method is idempotent, the request can be retried | The right column is a shortcut, not a definition: the protocol ties cacheability to individual codes (`200`, `203`, `204`, `206`, `300`, `301`, `308`, `404`, `405`, `410`, `414`, `501`), some members of the third class point at a stored copy rather than a new address (`304`), and some members of the fourth are open to a retry (`408`, `429`). The critical property of this table is that the right column uses **the class, not the code itself.** The intermediary does not need to know the difference between 404 and 400; both are in the fourth class, and in both, retrying is pointless. This is the highest level of detail a layer that does not know the application can know, and the protocol placed exactly that level in the first digit. The first class does not enter the model's measurement set, for a reason: 1xx is an **interim response**, it does not close the exchange. The intermediary neither caches it nor builds a decision on it; it forwards it as is and keeps waiting for the final response. Since what is measured is the decision taken per exchange, a response that does not close the exchange cannot enter the count. The resource design topic of the Web API Design course measures **which code should be chosen**: it counts how a loose mapping raises the number of requests. Here the choice is a given; what is measured is **how many decisions** a given class lets the intermediary make. #### Three Decisions The previous lesson's five decisions stay in place. The status line opens three more: | Decision | What determines the truth | |---|---| | `storeworthy` | Is the response actually successful | | `relocated` | Did the resource actually move to another address | | `retryworthy` | Did it fail on the server side, and is the method idempotent | The oracle's truth reads these from the response's **actual class**. The code the server declares is a separate field, and the two do not always agree: in the setup, a few server errors are declared as 200 — this lesson's form of the reading the course keeps returning to: nothing verifies what is written in the message. The decision set consists of forty exchanges and eight decisions — three hundred and twenty decisions. The smallest measurable difference is $1/320 = 0.0031$; no smaller difference is claimed. Assumptions of the measurement: - **HA7** — The previous lesson's forty exchanges and five decisions are taken as is; a status code is assigned on top with a separate generator, and the truth of the first five decisions does not change. - **HA8** — Status codes are drawn from a fixed list and do not enter the first class: an interim response does not close the exchange, and an exchange that does not close cannot enter the count. - **HA9** — The oracle's truth is the response's **actual class**; the code the server declares is a separate field, and in the setup some server errors are declared as 200. There is no mechanism that verifies the declaration. - **HA10** — Two regimes are compared: the intermediary either reads only the status line, or also opens the body. The only difference between them is where the class is read from — the code's first digit, or the body. - **HA11** — The body is a fixed-shape field representing resolved response content and carries the actual class correctly; the byte cost of opening the body comes from counting this field. - **HA12** — The round-trip balance is a count, not a timing measurement: the round trips added by the third class and removed by the fourth class are computed from the number of exchanges. #### Extending the Core The core is the same as the previous lesson's; on top of it an `assign_status` function, three new decisions and a second visibility tuple that also sees the body are added. The first five lines of the `oracle` function are untouched. ```python SEED = 20260809 SAFE = {"GET", "HEAD"} IDEMPOTENT = {"GET", "HEAD", "PUT", "DELETE"} METHOD_POOL = ["GET", "GET", "GET", "HEAD", "POST", "PUT", "DELETE"] PATH_POOL = ["/measurement/north", "/measurement/slope", "/summary", "/register", "/session"] STATUS_POOL = [200, 200, 200, 201, 204, 301, 400, 404, 500, 503, 503] BASE = ("storable", "shareable", "fresh", "repeatable", "redirectable") STATUS_DECISIONS = ("storeworthy", "relocated", "retryworthy") DECISIONS = BASE + STATUS_DECISIONS OPEN = ("method", "path", "host", "private_marker", "validator") LINE = OPEN + ("status",) LINE_BODY = LINE + ("body",) def generator(seed: int): d = seed % 2147483646 + 1 def r(n: int) -> int: nonlocal d d = (d * 48271) % 2147483647 return d % n return r def exchanges(count: int = 40, seed: int = SEED) -> list[dict]: r, out = generator(seed), [] for i in range(count): method, path = METHOD_POOL[r(7)], PATH_POOL[r(5)] private = path == "/session" or r(5) == 0 stale = r(3) == 0 out.append({ "no": i + 1, "method": method, "path": path, "private": private, "stale": stale, "private_marker": private and r(4) != 0, "tag_missed": stale and r(6) == 0, }) return out def assign_status(items: list[dict], seed: int = SEED + 101) -> list[dict]: r = generator(seed) for a in items: code = STATUS_POOL[r(11)] real = code // 100 misreported = real == 5 and r(3) == 0 # server declared the error as 200 a["actual_class"] = real a["status"] = 200 if misreported else code a["misreported"] = misreported return items def oracle(a: dict) -> dict: return { "storable": a["method"] in SAFE and not a["private"], "shareable": not a["private"], "fresh": not a["stale"], "repeatable": a["method"] in IDEMPOTENT, "redirectable": True, "storeworthy": a["actual_class"] == 2, "relocated": a["actual_class"] == 3, "retryworthy": a["actual_class"] == 5 and a["method"] in IDEMPOTENT, } def message(a: dict, visible: tuple) -> dict: full = {"method": a["method"], "path": a["path"], "host": "station.example", "private_marker": a["private_marker"], "validator": a["stale"] and not a["tag_missed"], "status": a["status"], "body": {"class": a["actual_class"], "path": a["path"], "detail": "resolved response body", "trace": f"trc-{a['no']:04d}"}} return {k: v for k, v in full.items() if k in visible} def byte_count(i: dict) -> int: return sum(len(k) + len(str(v)) + 4 for k, v in i.items()) def infer(i: dict) -> dict: k = {} if "method" in i: k["repeatable"] = i["method"] in IDEMPOTENT if "private_marker" in i: k["shareable"] = not i["private_marker"] if "method" in i: k["storable"] = i["method"] in SAFE and not i["private_marker"] if "validator" in i: k["fresh"] = not i["validator"] if "host" in i: k["redirectable"] = True if "status" in i: cls = i["body"]["class"] if "body" in i else i["status"] // 100 k["storeworthy"] = cls == 2 k["relocated"] = cls == 3 if "method" in i: k["retryworthy"] = cls == 5 and i["method"] in IDEMPOTENT return k def measure(batch: list[dict], visible: tuple, decisions: tuple = DECISIONS) -> tuple: d = y = e = b = 0 for x in batch: truth, given = oracle(x), infer(message(x, visible)) for decision in decisions: if decision not in given: e += 1 elif given[decision] == truth[decision]: d += 1 else: y += 1 b += byte_count(message(x, visible)) return d, y, e, b def distribution(batch: list[dict], field: str) -> dict: d: dict = {} for x in batch: d[x[field]] = d.get(x[field], 0) + 1 return dict(sorted(d.items())) ``` The single line inside `infer` carries the whole distinction: if the body is visible, the class is read from the body; if not, from `status // 100`. That single line is the only difference between the two intermediaries. #### Two Intermediaries ```python batch = assign_status(exchanges()) print("actual class distribution:", distribution(batch, "actual_class")) print("declared code distribution:", distribution(batch, "status")) print("responses misreporting their class:", sum(x["misreported"] for x in batch)) t = measure(batch, LINE, BASE) print(f"the base five decisions (same in both regimes): correct {t[0]}, wrong {t[1]}") print() print(f"{'regime':<14s} {'correct':>7s} {'wrong':>7s} {'unavailable':>12s} {'bytes':>6s}") for name, g in (("line", LINE), ("line+body", LINE_BODY)): d, y, e, b = measure(batch, g) print(f"{name:<14s} {d:7d} {y:7d} {e:12d} {b:6d}") ``` ``` actual class distribution: {2: 22, 3: 3, 4: 7, 5: 8} declared code distribution: {200: 16, 201: 5, 204: 4, 301: 3, 400: 4, 404: 3, 500: 1, 503: 4} responses misreporting their class: 3 the base five decisions (same in both regimes): correct 195, wrong 5 regime correct wrong unavailable bytes line 309 11 0 4381 line+body 315 5 0 8398 ``` The first column to read is **unavailable**, zero in both regimes. As long as the status line is visible, the intermediary gets all three new decisions; it does not ask the server and does not owe a round trip. The previous lesson's five decisions also stay in place: 195 correct, 5 wrong in both regimes. The whole difference sits in **six** of the three hundred and twenty decisions. The line regime makes 309 correct decisions, the regime that also opens the body makes 315. Against that, opening the body raises the byte count from 4381 to 8398. #### What the Body Buys Where the six decisions come from is worth a closer look. ```python print(f"{'':<20s} {'line':>14s} {'line+body':>16s}") print(f"{'decision':<20s} {'correct':>7s} {'wrong':>7s} {'correct':>8s} {'wrong':>7s}") for decision in STATUS_DECISIONS: s = [0, 0, 0, 0] for x in batch: truth = oracle(x) for j, g in enumerate((LINE, LINE_BODY)): v = infer(message(x, g)) s[2 * j + (0 if v[decision] == truth[decision] else 1)] += 1 print(f"{decision:<20s} {s[0]:7d} {s[1]:7d} {s[2]:8d} {s[3]:7d}") print() print(f"{'no':>3s} {'method':<7s} {'path':<20s} {'truth':>7s} {'declared':>9s} " f"{'idempotent':>10s}") for x in batch: if x["misreported"]: print(f"{x['no']:3d} {x['method']:<7s} {x['path']:<20s} {x['actual_class']:7d} " f"{x['status']:9d} {str(x['method'] in IDEMPOTENT):>10s}") ml, mb = measure(batch, LINE), measure(batch, LINE_BODY) gain, cost = mb[0] - ml[0], mb[3] - ml[3] print() print(f"decisions gained by the body {gain}, bytes it costs {cost}, " f"{cost / gain:.1f} bytes per decision") print("exchanges where the two regimes' decision sets diverge:", sum(1 for x in batch if infer(message(x, LINE)) != infer(message(x, LINE_BODY)))) added = sum(x["actual_class"] == 3 for x in batch) avoided = sum(x["actual_class"] == 4 for x in batch) print(f"round trips added by redirection {added}, round trips avoided by not " f"retrying pointlessly {avoided}, net over forty exchanges {added - avoided}") ``` ``` line line+body decision correct wrong correct wrong storeworthy 37 3 40 0 relocated 40 0 40 0 retryworthy 37 3 40 0 no method path truth declared idempotent 2 PUT /measurement/slope 5 200 True 8 GET /measurement/north 5 200 True 9 GET /summary 5 200 True decisions gained by the body 6, bytes it costs 4017, 669.5 bytes per decision exchanges where the two regimes' decision sets diverge: 3 round trips added by redirection 3, round trips avoided by not retrying pointlessly 7, net over forty exchanges -4 ``` In **thirty-seven** of the forty exchanges the two intermediaries produce the same decision set; opening the body changes not a single decision in these thirty-seven, only reading bytes. In the three exchanges where they diverge, six decisions are gained at a cost of 4017 bytes — 669.5 bytes per decision. This is also where the point of comparison appears. The status line hands the intermediary three decisions at once in all forty exchanges, costing only thirteen bytes per exchange. The body corrects **six** of three hundred and twenty decisions, costing ninety-seven extra bytes per exchange. Placing the class in the first digit is not a choice about saving space; it is a choice to put the smallest piece of data the intermediary needs at the very front of the response. #### A Decision Is Free, Acting on It Is a Round Trip The `relocated` decision is correct in forty of forty decisions in both regimes; it is this lesson's cheapest decision. Being cheap does not mean the decision's **consequence** is cheap too. A third-class response does not deliver the content requested, it only reports another address. The intermediary reads this from the message and asks no one — the decision itself needs no extra round trip. But carrying out what the decision calls for, going to the new address, is a full round trip. **Three** of the forty exchanges are third class, and these three add three round trips no header field could have absorbed; the only way to remove that round trip is to change the address. The fourth class works in the opposite direction. When the intermediary sees a 4xx, it knows resending the same request usually gives the same result, and does not send it. **Seven** of the forty exchanges are fourth class; in these seven, a single digit saves one round trip in advance that would not have changed the outcome anyway. The balance over forty exchanges is **minus four**: redirection adds three round trips, client error removes seven. The fifth class's balance depends on the method and so cannot be written on its own. What determines how many of the eight exchanges can be retried is not the status class but the method field measured in the previous lesson; the decision sits at the intersection of two fields. These three lines show why the measure is round trips, not the number of fields. The status line is a single field, and without touching the body at all it adds three round trips and removes seven, over forty exchanges. #### The Response That Misreports Its Class In three exchanges the server reported a server-side error as 200. The actual state is written in the body, not in the line. In the line regime these three responses break two decisions at once. `storeworthy` comes out wrong in all three: the intermediary puts a failed response into its cache and distributes the same error to later requests too. `retryworthy` also comes out wrong in all three, because all three use an idempotent method and, had the real class been known, the request could have been resent. `relocated` is not broken: a wrong declaration does not touch the third class. The real conclusion: **the intermediary did not make a wrong decision; it responded correctly to a wrong declaration.** Reading the first digit comes bundled with the assumption that the first digit was written correctly. No mechanism in the protocol tests this assumption, and opening the body is not a fix either, since the body is written by the same server. There is a narrowing side here too. When the server writes the status line correctly, the intermediary loses nothing; when it writes it wrong, closing the gap with the body costs ninety-seven bytes per exchange, and in thirty-seven of the forty exchanges that byte cost is spent for nothing. Reporting the class correctly is the single thing that minimizes the data the intermediary has to read. #### Summary - The status line's first digit gives a status class; the intermediary draws its caching, routing and retry decisions from this single digit, not from the whole code. - Class-based reading does not require the intermediary to know the code's detail: 400 and 404 carry the same instruction. - On the set of forty exchanges and eight decisions, the line regime gives 309 correct, 11 wrong decisions and 4381 bytes; the regime that also opens the body gives 315 correct, 5 wrong decisions and 8398 bytes. - The two regimes produce the same decision set in thirty-seven of the forty exchanges; the six decisions the body buys cost 669.5 bytes each. - A cheap decision does not mean a cheap consequence: redirection adds three round trips over forty exchanges, client error removes seven, and the balance is minus four. - The three responses that misreport their class break the `storeworthy` and `retryworthy` decisions; the intermediary does not make a wrong decision, it responds correctly to a wrong declaration. #### Next Step The status line says **what** the response is. There is something it does not say: the same address can have more than one counterpart, and the server does not report in the status line which one it sent. The same measurement record can be served in separate formats, separate languages, separate encodings; either one comes back with a 200. The next lesson measures the place where this choice is made — the header fields — and asks two questions at once: does adding irrelevant fields buy the intermediary any decisions, and if the field the choice was based on is not reported, which copy does the intermediary give to whom. ### Headers and Content Negotiation Source: https://academia.sh/en/courses/application-protocols/headers-and-content-negotiation Adding a field does not buy a decision: three new headers raise the byte count by 79 percent for zero decisions, while declaring the field that enters negotiation fixes nine wrong copies. The previous lesson measured the response's first line and left one thing unsaid: the status line says **what** the response is, not which counterpart was sent. The same measurement record can be served in separate formats, separate languages, separate compressions, and all of them come back with a 200. The place where that choice is made is the header fields. This lesson asks two questions at once. First: does adding a field to the message buy the intermediary a decision. Second: when the same address has more than one counterpart, and the field the choice was based on is not declared, which copy does the intermediary give to whom. #### Two Kinds of Header Field A header field is a name and a value separated by a colon. The dump below is for teaching purposes and is not run: ```text GET /measurement/north HTTP/1.1 Host: station.example Accept: application/json Accept-Language: tr X-Tracking-Id: trc-0004 ``` For the intermediary, these five lines split into **two sets**. `Host` establishes its routing decision; the `Accept` family determines which counterpart the server produces. The last line is a field the intermediary cannot resolve: it reads the name, it reads the value, but it cannot draw a decision from it. This split is the direct counterpart of the course's measure. **A field the intermediary cannot resolve is nothing more than bytes to it.** That sentence is an intuition; this lesson's first measurement turns it into a number. #### Representation and Negotiation An address names a **resource**; the resource's counterpart as carried over the network is called a **representation**. The same resource can have more than one representation: the same measurement record can be delivered as a structured format, as comma-separated lines, or as plain text. The client declares which one it wants through the `Accept` family of fields; the server looks at that declaration and selects a representation. This selection is called **content negotiation**. It has three axes, and all three use the same mechanism: | Axis | Client's declaration | What is selected | |---|---|---| | Format | `Accept` | Media type | | Language | `Accept-Language` | The text's language | | Encoding | `Accept-Encoding` | Compression in transit | The consequence negotiation has for the intermediary is this: **a single address no longer corresponds to a single response.** When the intermediary stores a copy, it also has to know which request that copy was produced for. If it does not know, it hands the stored copy to the wrong client. #### The Requester's Declaration, the Server's Declaration Negotiation runs on two declarations, and they point in separate directions. The request-side `Accept` family lists what the client **can accept**; it is a preference list, not a single choice. The response-side `Content-Type` declares what the server **actually sent**, and it is singular. The intermediary needs both, for separate reasons. The request-side declaration lets it test whether the copy it holds matches this request. The response-side declaration lets it know what the copy it stored actually is. Neither works without the other: knowing what the client wants does not build a match without knowing what is on hand. The two also fail differently. Because `Accept` is a preference list, the server may be unable to produce any option on it; negotiation then comes back empty, and the decision returns to the client. Because `Content-Type` is a single value, it can be written wrong: when the server sends one format but declares another, the intermediary stores it under the wrong declaration and binds the copy to the wrong key. Encoding parts ways with the other two axes at one point. Format and language touch the representation's **meaning**: a different format is a different record layout, a different language is a different text. Encoding leaves the meaning untouched and changes only the transport-level representation — a compressed record, decompressed, is identical to its uncompressed counterpart. The intermediary still cannot ignore it: handing a compressed copy to a client that never declared it could accept compression produces a response that client cannot read. Meaning stays the same; the key still has to be kept separate. Assumptions of the two measurements: - **HA13** — The same forty exchanges and five decisions are used; a separate generator assigns the format and language each exchange requests. `oracle` and `infer` are untouched. - **HA14** — The three added fields carry real information but never enter a branch in the intermediary's code; what is measured is not a field's size but whether the reader can branch on it. - **HA15** — The second measurement runs only over **storable** exchanges: safe method, not personalized. The rest of the forty never enter the intermediary's store. - **HA16** — The intermediary stores one copy per key, and only the **declared** fields join the address in that key; the first copy to arrive holds its place and is given out to every later request. - **HA17** — The representation the client wants is known because we generated it ourselves; the copy handed out is counted right or wrong against this known request. - **HA18** — The declared vary is tried at three values: none, format only, format and language. No mechanism tests the declaration's accuracy; the intermediary believes whatever fields the server lists. #### First Measurement: Adding a Field The first measurement takes the five-decision set built by the previous two lessons as is and changes exactly one thing: three fields the intermediary cannot resolve are added to the message. The `oracle` and `infer` functions are untouched. ```python SEED = 20260809 SAFE = {"GET", "HEAD"} IDEMPOTENT = {"GET", "HEAD", "PUT", "DELETE"} METHOD_POOL = ["GET", "GET", "GET", "HEAD", "POST", "PUT", "DELETE"] PATH_POOL = ["/measurement/north", "/measurement/slope", "/summary", "/register", "/session"] FORMAT_POOL = ["json", "json", "csv", "text", "json"] LANGUAGE_POOL = ["tr", "tr", "en", "tr", "en"] DECISIONS = ("storable", "shareable", "fresh", "repeatable", "redirectable") VISIBLE = {"open": ("method", "path", "host", "private_marker", "validator")} VISIBLE["open+extra"] = VISIBLE["open"] + ("tracking_id", "client_version", "region") def generator(seed: int): d = seed % 2147483646 + 1 def r(n: int) -> int: nonlocal d d = (d * 48271) % 2147483647 return d % n return r def exchanges(count: int = 40, seed: int = SEED) -> list[dict]: r, out = generator(seed), [] for i in range(count): method, path = METHOD_POOL[r(7)], PATH_POOL[r(5)] private = path == "/session" or r(5) == 0 stale = r(3) == 0 out.append({ "num": i + 1, "method": method, "path": path, "private": private, "stale": stale, "private_marker": private and r(4) != 0, "tag_missed": stale and r(6) == 0, }) return out def assign_preferences(items: list[dict], seed: int = SEED + 211) -> list[dict]: r = generator(seed) for a in items: a["format"], a["language"] = FORMAT_POOL[r(5)], LANGUAGE_POOL[r(5)] return items def oracle(a: dict) -> dict: return { "storable": a["method"] in SAFE and not a["private"], "shareable": not a["private"], "fresh": not a["stale"], "repeatable": a["method"] in IDEMPOTENT, "redirectable": True, } def message(a: dict, visible: tuple) -> dict: full = {"method": a["method"], "path": a["path"], "host": "station.example", "private_marker": a["private_marker"], "validator": a["stale"] and not a["tag_missed"], "tracking_id": f"trc-{a['num']:04d}", "client_version": "measure-client", "region": "north-slope"} return {k: v for k, v in full.items() if k in visible} def byte_count(i: dict) -> int: return sum(len(k) + len(str(v)) + 4 for k, v in i.items()) def infer(i: dict) -> dict: k = {} if "method" in i: k["repeatable"] = i["method"] in IDEMPOTENT if "private_marker" in i: k["shareable"] = not i["private_marker"] if "method" in i: k["storable"] = i["method"] in SAFE and not i["private_marker"] if "validator" in i: k["fresh"] = not i["validator"] if "host" in i: k["redirectable"] = True return k def measure(label: str, batch: list[dict]) -> tuple: d = y = e = t = b = 0 for a in batch: truth, given = oracle(a), infer(message(a, VISIBLE[label])) missing = False for decision in DECISIONS: if decision not in given: e += 1 missing = True elif given[decision] == truth[decision]: d += 1 else: y += 1 t += 1 if missing else 0 b += byte_count(message(a, VISIBLE[label])) return d, y, e, t, b def negotiate(batch: list[dict], vary: tuple) -> tuple: """An intermediary that stores one copy per address; only the declared fields enter the key.""" store, correct, wrong, vary_bytes = {}, 0, 0, 0 for a in batch: if not (a["method"] in SAFE and not a["private"]): continue vary_bytes += byte_count({"vary": vary}) if vary else 0 key = (a["path"],) + tuple(a[field] for field in vary) requested = (a["format"], a["language"]) served = store.setdefault(key, requested) if served == requested: correct += 1 else: wrong += 1 return correct, wrong, len(store), vary_bytes ``` The three added fields are a tracking ID, a client version, and a region name. All three carry real information, and none of the three enters a branch in the `infer` function. ```python batch = assign_preferences(exchanges()) print(f"{'visibility':<10s} {'correct':>7s} {'wrong':>7s} {'unavailable':>11s} " f"{'extra round':>13s} {'bytes':>6s}") for label in ("open", "open+extra"): d, y, e, t, b = measure(label, batch) print(f"{label:<10s} {d:7d} {y:7d} {e:11d} {t:13d} {b:6d}") ba, be = measure("open", batch)[4], measure("open+extra", batch)[4] print(f"fields {len(VISIBLE['open'])} -> {len(VISIBLE['open+extra'])}, " f"bytes {ba} -> {be}, increase {100 * (be - ba) / ba:.0f}%, decisions gained " f"{measure('open+extra', batch)[0] - measure('open', batch)[0]}") ``` ``` visibility correct wrong unavailable extra round bytes open 195 5 0 0 3861 open+extra 195 5 0 0 6901 fields 5 -> 8, bytes 3861 -> 6901, increase 79%, decisions gained 0 ``` All four columns of the decision table stay the same: 195 correct, 5 wrong, 0 unavailable, 0 extra round trips. Bytes, though, rise from 3861 to 6901 — **a 79 percent increase, zero decisions.** Three thousand and forty extra bytes were carried across the forty exchanges, and nothing was gained in return. This is the course's second claim, and its price is paid here: **adding a field does not buy a decision.** A field's value comes not from the information it carries but from whether the reader can build a branch on that information. A tracking ID may be valuable to the measurement station; to the intermediary it is nine hundred and twenty bytes across the forty exchanges, and nothing more. #### Second Measurement: The Cost of Negotiation The second measurement runs in the opposite direction. Now the added field is one the intermediary **can** resolve, and the question is: how many decisions does declaring which field entered negotiation buy. The intermediary binds the copy it stores to a key. Without a declaration, the key is just the address: the first copy to arrive is stored and given out to every later request. With a declaration, the declared fields also enter the key, and every representation is stored separately. The measurement runs only over storable exchanges — safe method, not personalized. This declaration's protocol counterpart is the `Vary` header field: the server lists which header fields it looked at while producing the response. The model's `vary` tuple is exactly this list. The field sits on the response side for a reason — only the party making the selection knows what entered negotiation, and the intermediary knows it only if told. The list's length has a limit, visible in the measurement's last column. Every field that enters the key multiplies the number of stored copies. If a field whose value differs for every client enters the list, every request produces a separate key, and the intermediary never reuses a single copy. The correct declaration lists exactly the fields that entered negotiation and nothing else: too little produces a wrong copy, too much produces a copy that is never reused. ```python stored = [a for a in batch if a["method"] in SAFE and not a["private"]] print(f"exchanges entering negotiation {len(stored)}, " f"requested representations {len({(a['format'], a['language']) for a in stored})} kinds, " f"addresses {len({a['path'] for a in stored})}") print() print(f"{'declared vary':<22s} {'correct copy':>12s} {'wrong copy':>11s} " f"{'stored':>7s} {'vary bytes':>10s}") for vary in ((), ("format",), ("format", "language")): d, y, n, vb = negotiate(batch, vary) print(f"{str(vary):<22s} {d:12d} {y:11d} {n:7d} {vb:10d}") d0, y0, n0, b0 = negotiate(batch, ()) d2, y2, n2, b2 = negotiate(batch, ("format", "language")) print() print(f"full vary: decisions gained {d2 - d0}, bytes spent {b2 - b0}, " f"{(b2 - b0) / (d2 - d0):.1f} bytes per decision; " f"stored copies {n0} -> {n2}") ``` ``` exchanges entering negotiation 13, requested representations 6 kinds, addresses 4 declared vary correct copy wrong copy stored vary bytes () 4 9 4 0 ('format',) 9 4 8 247 ('format', 'language') 13 0 11 390 full vary: decisions gained 9, bytes spent 390, 43.3 bytes per decision; stored copies 4 -> 11 ``` There are thirteen storable exchanges, four addresses, and six kinds of requested representation. The measurement band's lower bound in this set is $1/13 = 0.077$; no difference smaller than that is claimed. With no declaration at all, the intermediary stores four copies and **hands out the wrong copy to nine of the thirteen requests.** A wrong copy is not a broken response: it is a valid, fresh representation from the correct address — it just does not belong to the requester. A requester expecting plain text gets a structured record; one expecting Turkish gets English. When only format is declared, wrong copies drop from nine to four. The remaining four come from the language axis: the intermediary has learned to separate by format, not by language. **An incomplete declaration is a smaller version of no declaration at all**, because missing even a single axis is enough to make a copy wrong. When both axes are declared, wrong copies drop to zero. #### Reading the Two Measurements Side by Side Both measurements add a field to the same message, and their outcomes come out diametrically opposed. | Added | Bytes | Decisions gained | |---|---|---| | Three unresolvable fields | 3040 | 0 | | Declared vary | 390 | 9 | The difference is not in the fields' size but in what the reader can do with them. The three fields in the first measurement add no branch to the intermediary's code at all; the single field in the second changes its key. This is why the measure has to be **decisions**, not **field count**: field count would have counted both as "something was added to the message." Negotiation has a second cost, and it is not measured in bytes. Once the declaration is complete, the number of copies the intermediary stores rises from four to **eleven**. The same address is no longer a single entry for the intermediary; each representation takes its own slot and is refreshed on its own. The price of handing out the right copy is storing the same resource more than once. The fourth reading holds here too. The intermediary cannot test which fields actually entered negotiation. If the server under-declares, it stores under an incomplete key and distributes the wrong copy **without raising a single error.** No mechanism checks the declaration's accuracy; the intermediary believes it. Every gain and cost counted in this lesson rests on that single assumption. #### Summary - Header fields split into two groups as far as the intermediary is concerned: ones that resolve into a decision, and ones that are only carried. - On the set of forty exchanges, adding three unresolvable fields leaves the decision table unchanged — 195 correct, 5 wrong — but bytes rise from 3861 to 6901, a 79 percent increase. - An address names a resource; content negotiation is the same resource having more than one representation across the format, language, and encoding axes. - If the field entering negotiation is not declared, the intermediary hands out the wrong copy in nine of thirteen storable exchanges; declaring format alone leaves four, declaring both axes leaves zero. - Added to the same message, 3040 bytes buy zero decisions and 390 bytes buy nine; this is why the measure is decisions, not field count. - The price of handing out the right copy is not limited to bytes: the number of stored copies rises from four to eleven. #### Next Step Content negotiation means the same address has **more than one representation**, and the intermediary derives which copy goes to whom from the header fields. Every representation measured in this lesson shared one property: two clients making the same declaration deserved the same copy. Format and language describe not **who** the requester is, but what they want. So what happens when the response is **personalized** — when two clients arriving at the same address with the same format and language declaration deserve different responses? Then the field needed to select the right copy has to carry not what was requested but who it belongs to. The next lesson takes up that field and the intermediary's relationship with it. ### Cookies and State Transfer Source: https://academia.sh/en/courses/application-protocols/cookies-and-state-transfer A cookie's scope rule determines which messages it attaches to, its flag rule determines who the stored response goes to; across forty exchanges, three scope rules carry the cookie on 11, 24 and 40 messages in turn, and two exchanges left unflagged cost the intermediary two wrong shareability decisions. The previous lesson set up content negotiation: the same address corresponds to more than one representation, and the intermediary derives which copy goes to whom from the header fields. Negotiation selects a representation by the request's **format preference** — language, encoding, media type — and reads every one of these directly from the request. So what if the response is personalized not to format but to a **person**? Then the selecting field has to carry an identity, not a preference. A **cookie** is the field that carries the personalization itself. This lesson's question is not what a cookie is for, but what the intermediary knows when it sees one, and what it cannot know when it does not. #### State in a Stateless Protocol HTTP is defined as **stateless**: every request has to be understandable on its own, and the server is not required to remember the previous one. A cookie does not break this definition; it works around it. The server does not keep state itself — it **keeps it at the client**, which sends it back with every request. Each request stays understandable on its own after all, because the state now sits inside the message. The exchange comes down to two fields. In its response, the server declares a name–value pair and its attributes through `Set-Cookie`; in later requests, the client sends the name–value pair back through `Cookie`. The attributes travel out and not back: the client sends only the pair. ```text # teaching dump, not run server -> client HTTP/1.1 200 OK Set-Cookie: session=k7; Path=/session; Domain=station.example; Max-Age=3600; HttpOnly; SameSite=Strict; Secure Cache-Control: private Vary: Cookie client -> server GET /session/summary HTTP/1.1 Host: station.example Cookie: session=k7 ``` Each attribute answers one question. `Domain` and `Path` set the cookie's **scope**: which host and path prefix it is sent for. `Max-Age` sets its lifetime; left unset, the cookie is a session cookie and drops when the client closes. `HttpOnly` closes it off from scripts inside the document. `SameSite` determines whether it travels on requests triggered from another origin. `Secure` ties it to the type of transport. Two response-side fields concern the intermediary directly. `Cache-Control: private` declares that the response must not go into a shared cache. `Vary: Cookie` declares which request field the stored copy depends on — the same address has separate copies keyed by `Cookie`. The model's **personalization flag** is exactly what these two fields declare. The authentication flow itself is not this course's subject; logging in, session renewal, and token use belong to the **Authentication and Authorization** course. Here the cookie is measured as a scope rule, not an identity mechanism. #### What the Cookie's Value Tells the Intermediary A cookie is a name–value pair, and only the server knows what the value means. To the intermediary, the string `session=k7` is just a string: it does not say whose session it is, how much privilege it carries, or whether it is a session token or a language preference. Decoding the cookie's value would require knowing the application, and the intermediary does not. This has a direct consequence: **the intermediary cannot infer that a response is personalized just because a cookie is present.** If it could, it would give up caching a shareable response over a cookie carrying nothing but a language preference. The reverse fails too: no cookie does not mean the response is shareable, since the server may have read identity from some other field. The decision lives not in the cookie, therefore, but **in the response's flag**. Whatever the server declares, the intermediary knows; whatever it does not declare, it does not know. The model's personalization flag is exactly this declaration, and the only sign of personalization visible in the message. The cookie itself adds no decision to the measurement; what enters is **which messages the cookie is present in**, not what it carries. The lifetime attribute falls into the same class: the difference between a cookie with `Max-Age` set and a session cookie is how long the client keeps it, but the decision drawn from that message is the same either way. The intermediary does not know, and does not need to know, when the cookie will drop. #### Two Separate Rules Two decisions concern a cookie, and confusing them is the most common mistake. The **scope rule** concerns the request: **which messages does the cookie travel on?** `Domain` and `Path` set this, and the client decides. The **flag rule** concerns the response: **who does the stored response go to?** `Cache-Control` and `Vary` set this, the server declares it, and the intermediary carries it out. The two are independent. A correctly scoped cookie can travel alongside a response with no flag set — it reaches the right place, but the response can go to the wrong person. The reverse happens too: a correctly flagged response can travel alongside a cookie scoped wider than it needs to be. Assumptions of the measurement: - **HA41** — The forty exchanges come from the shared setup; the oracle is known because we generated it ourselves, and says whether each response is actually personalized. - **HA42** — Three scope rules are tried: the session path alone, the measurement-and-session prefix, the whole host. A scope rule looks only at the path; it does not use the oracle. - **HA43** — The personalization flag is the only declaration visible in the message. The server skips setting it on some personalized responses; **nothing verifies what is written in the message.** - **HA44** — When the intermediary sees the flag, it makes the `shareable` and `storable` decisions without an extra round trip; when it does not, it makes neither and owes the server one more round trip. - **HA45** — The byte count comes from field name, value and separator length; this is a count of the message's content, not a network measurement. #### The Measurement ```python """Cookie scope and state transfer: the decision the intermediary draws from the private flag. Part 1 - which messages the scope rule attaches the cookie to. Part 2 - the decision table with the flag visible, and folded into the envelope. """ SEED = 20260809 SAFE = {"GET", "HEAD"} IDEMPOTENT = {"GET", "HEAD", "PUT", "DELETE"} METHOD_POOL = ["GET", "GET", "GET", "HEAD", "POST", "PUT", "DELETE"] PATH_POOL = ["/measurement/north", "/measurement/slope", "/summary", "/register", "/session"] DECISIONS = ("storable", "shareable", "fresh", "repeatable", "redirectable") def generator(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def exchanges(count=40): r, out = generator(SEED), [] for i in range(count): method, path = METHOD_POOL[r(7)], PATH_POOL[r(5)] private = path == "/session" or r(5) == 0 stale = r(3) == 0 out.append({"num": i + 1, "method": method, "path": path, "private": private, "stale": stale, "private_marker": private and r(4) != 0, "tag_missed": stale and r(6) == 0}) return out def oracle(a): return {"storable": a["method"] in SAFE and not a["private"], "shareable": not a["private"], "fresh": not a["stale"], "repeatable": a["method"] in IDEMPOTENT, "redirectable": True} def message(a, visible): full = {"method": a["method"], "path": a["path"], "host": "station.example", "private_marker": a["private_marker"], "validator": a["stale"] and not a["tag_missed"]} return {k: v for k, v in full.items() if k in visible} def byte_count(i): return sum(len(k) + len(str(v)) + 4 for k, v in i.items()) def infer(i): k = {} if "method" in i: k["repeatable"] = i["method"] in IDEMPOTENT if "private_marker" in i: k["shareable"] = not i["private_marker"] if "method" in i: k["storable"] = i["method"] in SAFE and not i["private_marker"] if "validator" in i: k["fresh"] = not i["validator"] if "host" in i: k["redirectable"] = True return k VISIBLE = {"open": ("method", "path", "host", "private_marker", "validator"), "in envelope": ("method", "path", "host", "validator")} SCOPE = {"session path": lambda a: a["path"] == "/session", "measurement prefix": lambda a: a["path"].startswith(("/measurement", "/session")), "whole host": lambda a: True} batch = exchanges() print(f"exchanges {len(batch)} | decisions {len(batch) * len(DECISIONS)} | personalized " f"{sum(a['private'] for a in batch)} | left unflagged " f"{sum(a['private'] and not a['private_marker'] for a in batch)}") print("exchanges left unflagged:", [(a["num"], a["method"], a["path"]) for a in batch if a["private"] and not a["private_marker"]]) print() print(f"{'scope rule':<19s} {'carries':>9s} {'priv+carries':>13s} " f"{'priv, no cookie':>16s} {'needless carry':>15s}") for name, rule in SCOPE.items(): print(f" {name:17s} {sum(rule(a) for a in batch):9d}" f" {sum(rule(a) and a['private'] for a in batch):13d}" f" {sum(not rule(a) and a['private'] for a in batch):16d}" f" {sum(rule(a) and not a['private'] for a in batch):15d}") print() print("decision open: correct wrong in envelope: correct wrong unavailable") for decision in DECISIONS: s = {} for name in VISIBLE: d = y = e = 0 for a in batch: g, v = oracle(a), infer(message(a, VISIBLE[name])) if decision not in v: e += 1 elif v[decision] == g[decision]: d += 1 else: y += 1 s[name] = (d, y, e) print(f" {decision:18s} {s['open'][0]:5d} {s['open'][1]:6d}" f" {s['in envelope'][0]:16d} {s['in envelope'][1]:6d}" f" {s['in envelope'][2]:10d}") print() print("visibility correct wrong unavailable extra round bytes") for name in VISIBLE: d = y = e = t = b = 0 for a in batch: g, v = oracle(a), infer(message(a, VISIBLE[name])) missing = False for decision in DECISIONS: if decision not in v: e += 1 missing = True elif v[decision] == g[decision]: d += 1 else: y += 1 t += 1 if missing else 0 b += byte_count(message(a, VISIBLE[name])) print(f" {name:12s} {d:5d} {y:7d} {e:11d} {t:13d} {b:6d}") ``` ``` exchanges 40 | decisions 200 | personalized 17 | left unflagged 2 exchanges left unflagged: [(1, 'POST', '/register'), (34, 'POST', '/register')] scope rule carries priv+carries priv, no cookie needless carry session path 11 11 6 0 measurement prefix 24 12 5 12 whole host 40 17 0 23 decision open: correct wrong in envelope: correct wrong unavailable storable 40 0 0 0 40 shareable 38 2 0 0 40 fresh 37 3 37 3 0 repeatable 40 0 40 0 0 redirectable 40 0 40 0 0 visibility correct wrong unavailable extra round bytes open 195 5 0 0 3861 in envelope 117 3 80 40 2956 ``` #### What Scope Costs The upper table places the three scope rules side by side, and none is free of trouble. The rule **narrowed to the session path** carries the cookie on **11** messages, and all eleven are genuinely personalized; needless carrying is **0**. Its price sits in the column to the right: **6** personalized responses fall outside the scope. The setup says this directly — personalization does not come from the path itself; the server also produces some personalized responses outside `/session`. The rule **opened to the whole host** closes those six: personalized responses left outside scope drop to **0**. In exchange, the cookie is carried needlessly on **23** messages — on more than half of the forty exchanges, for no purpose at all. The **measurement prefix** rule sits between the two: **12** needless carries and **5** personalized responses left outside scope. The pattern: **the scope rule looks at the path, and personalization does not stay confined to a path.** Widening the rule reduces what falls outside scope but grows needless carrying linearly. No path zeroes out both at once, because the address is the only thing a scope rule can know. #### What the Flag Costs The lower tables look at the second rule. In **17** of the forty exchanges the response is personalized, and the server skipped setting the flag on **2** of those; the intermediary takes both for shareable, so the `shareable` row reads **38** correct, **2** wrong. The `storable` row reads **40** correct, and that is not a coincidence. Both exchanges left unflagged are `POST /register`, and that method is already not safe. A missing flag only costs this decision on a response that would **otherwise be storable**; no exchange here meets that condition, so the row stays clean. Had the same gap occurred on a safe-method exchange, the wrong count would have risen from two to four. Two wrong decisions, in a set of two hundred, is a share of **0.0100**. The smallest measurable difference in this set is **1/200 = 0.0050**; two wrong decisions is twice that and sits inside the measurement band. In the last table, the open regime reads **195** correct, **5** wrong, **0** unavailable. **2** of the five wrong decisions come from the missing flag measured here; the remaining three sit in the `fresh` row and come from an entirely different field, the validator tag, not measured in this lesson. #### Misconfigured Scope and Its Narrowings A broadly configured scope is not an attack, it is a **configuration flaw**: the rule is written, the syntax is correct, the cookie works, and nothing signals a problem. Three surfaces follow, each with the field that closes it. **First surface — needless carrying.** With scope opened to the whole host, the cookie is carried on **23** messages for no purpose at all. **Narrowing:** pulling `Path` back to the session path drops needless carrying to **0**; its price is the **6** personalized responses left outside scope. Leaving `Domain` unset ties scope to a single host; setting it opens scope to subdomains. `SameSite` narrows scope by **the context that triggered the request** rather than by path, closing a direction `Path` cannot: a request reaching the same path but from another origin. **Second surface — a stored response with no flag.** In two exchanges the intermediary makes the wrong `shareable` decision; putting such a response into a shared cache means handing a personalized copy to a different request. **Narrowing:** setting `Cache-Control: private` corrects the decision; `Vary: Cookie` binds the stored copy to the `Cookie` field and keeps the same address's copies separate. Both are the server's declaration, and when they are not set, the intermediary has no way to know. **Third surface — the intermediary seeing the flag.** The intermediary reads which message is personalized; that is what lets it decide without a round trip. **Narrowing:** folding the flag field into the transport envelope hides it from the intermediary. The `in envelope` row gives the price: **80** decisions move from the middle to the endpoints, all forty exchanges owe one more round trip each, and bytes drop from **3861** to **2956**. The gain is that wrong decisions drop from **5** to **3** — a decision the intermediary cannot make is one it cannot get wrong. #### Summary - A cookie carries state in a stateless protocol by keeping the state at the client; attributes travel out with `Set-Cookie`, and only the name–value pair travels back, in `Cookie`. - The scope rule determines **which message** carries the cookie, the flag rule determines **who** the stored response goes to; the two are independent and each can be misconfigured on its own. - The three scope rules carry the cookie on **11**, **24**, and **40** messages in turn; as scope widens, personalized responses left outside it drop from **6** to **0**, and needless carrying rises from **0** to **23**. - **17** of the forty exchanges are personalized, and **2** have no flag set; the intermediary makes two wrong `shareable` decisions, while `storable` stays clean because those two already use a method that is not safe. - Folding the flag field into the transport envelope drops wrong decisions from **5** to **3**, but moves **80** decisions to the endpoints and creates a debt of forty round trips; that is the price of hiding it. #### Next Step A cookie is a header field, and it is resent on **every** request within its scope. Across forty exchanges, the host name repeats forty times, the method repeats forty times, and the cookie repeats as many times as its scope reaches — all over the same connection, all written out from scratch. The next lesson looks at how this repetition gets packed: what the move from a text line to a binary frame buys the intermediary, which field header compression turns unreadable for it, and where the decision tied to an unread field lands. ### HTTP Versions Source: https://academia.sh/en/courses/application-protocols/http-versions The same fields are measured across three framing regimes: the text line takes 3861 bytes and 251 blocking units, the binary frame that keeps context gives the same outcome on all 200 decisions for 459 bytes, and in the binary frame with no context 189 decisions move to the endpoints. A cookie is a header field, and it is resent on every request within its scope. So is the host name, so is the method, so are the preferences the client declares. In a forty-exchange session, the same strings cross the wire forty times, written out from scratch each time. This repetition is the result of a design decision: an HTTP message is built from **text lines**, and a text line is self-sufficient — the parser reads it, finds the colon, separates name from value. The difference between versions is this lesson's subject, and it is not in meaning but in **framing**: which fields travel does not change, how they get packed onto the wire does. Our question — what does changing the packing buy the intermediary in between, and what does it cost? #### Same Meaning, Different Framing HTTP/1.1 writes a message line by line. The request line carries the method and path, each following line is a header field, and a blank line announces that the body starts. HTTP/2 and HTTP/3 put the same fields into **binary frames**: every frame has a type and a length, and header fields are compressed with a separate encoding. An HTTP/2 frame also carries a stream identifier and flags; HTTP/3 has neither of those fields, because the transport itself separates the streams. ```text # teaching dump, not run # text-line framing GET /measurement/north HTTP/1.1 Host: station.example Cookie: session=k7 Accept: text/csv Connection: keep-alive # binary framing, same fields FRAME type=HEADERS stream=7 flags=END_HEADERS :method -> index :path -> literal /measurement/north :authority -> index cookie -> index accept -> index FRAME type=DATA stream=7 flags=END_STREAM ``` The two forms say the same thing. Method, path, host, and the two preference fields appear in both; `:method` and `:path` are the counterparts of the text-form request line. Three things change: fields travel in **frames, not lines**, frames carry a **stream identifier**, and header fields can be sent **by index**. The stream identifier makes **multiplexing** possible: more than one exchange can advance interleaved over a single connection, because every frame carries which stream it belongs to. Text-line framing has no such field; requests on one connection can only proceed one after another, and this holds for a **persistent connection** too — the connection stays open, but order is preserved. HTTP/3 changes not the framing but the **transport**. The frame layout stays similar; the transport layer underneath offers independent streams instead of a single ordered byte stream. Why this matters is the next section's subject. #### Head-of-Line Blocking in Two Layers **Head-of-line blocking** is what happens when the item at the front of a queue cannot advance, so everything behind it waits too. HTTP has two separate forms of this, and confusing them makes the difference across versions invisible. **The application-layer form:** if requests on one connection are sequential, a slow response holds up every exchange behind it. The slowness does not come from the network — the server is assembling that response from more than one record. Multiplexing removes this form. **The transport-layer form:** if the transport layer delivers in order, a lost segment holds up everything that follows it, regardless of which stream it belongs to, because the transport layer does not know about streams. Multiplexing does **not** remove this form; it only moves it down from the layer above. What removes it is the transport itself becoming stream-aware. This distinction is a direct consequence of the byte-stream abstraction established in the **Network Models and Protocols** course: once an ordered, lossless stream is promised, keeping that promise requires a queue that waits. The introductory intuition from **How the Internet Works** ties to a number here. Blocking carries a separate meaning for the intermediary, which does not only read a message but **forwards** it — that too is a queue. If application order is preserved, the intermediary cannot send a response it holds before its turn comes, and it holds up everything behind it while waiting for one response. What multiplexing removes is not the server's queue but the intermediary's. #### Header Compression and the Intermediary's Context Binary framing sends header fields by index: a name–value table is kept for the connection's duration, and when a pair recurs, what is sent is not its name and value but **its position in the table**. The gain is large, since fields like the host name and the method almost never change. The price: **an index is meaningless to whoever does not keep the table.** An intermediary present since the connection began builds the same table and resolves every field. One that joined later, or forwards frames without decoding them, can read only the fields sent literally. - **HA46** — The same forty exchanges, five fields, and oracle are used; only how the fields are written on the wire changes. - **HA47** — In text-line framing, every field is written out from scratch in every message: name, value, separator. - **HA48** — In binary framing, a name–value pair is written literally the first time it appears and enters the table; later occurrences send a **1-byte** index. - **HA49** — An intermediary that keeps the table resolves every field. One that does not reads only fields sent literally; an indexed field does not exist for it. - **HA50** — The exchange slow on the application side is `/summary`; the server assembles that response from more than one record. - **HA51** — The exchange that loses data in transport is chosen by a separate generator, with a single modulus. - **HA52** — A blocking unit is a count, not a duration: an exchange waits one unit behind every blocker ahead of it in the same queue. - **HA53** — In parallel connections, exchanges are distributed to connections in turn, and **each connection has its own table.** - **HA54** — The measurement is not a network measurement; the byte count is a count of the message's content. #### The Measurement ```python """HTTP versions: what framing buys the intermediary, and what it costs. Part 1 - the same fields across three framing regimes: decisions and bytes. Part 2 - head-of-line blocking and multiplexing: blocked exchanges. """ SEED = 20260809 SAFE = {"GET", "HEAD"} IDEMPOTENT = {"GET", "HEAD", "PUT", "DELETE"} METHOD_POOL = ["GET", "GET", "GET", "HEAD", "POST", "PUT", "DELETE"] PATH_POOL = ["/measurement/north", "/measurement/slope", "/summary", "/register", "/session"] DECISIONS = ("storable", "shareable", "fresh", "repeatable", "redirectable") FIELDS = ("method", "path", "host", "private_marker", "validator") def generator(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def exchanges(count=40): r, out = generator(SEED), [] for i in range(count): method, path = METHOD_POOL[r(7)], PATH_POOL[r(5)] private = path == "/session" or r(5) == 0 stale = r(3) == 0 out.append({"num": i + 1, "method": method, "path": path, "private": private, "stale": stale, "private_marker": private and r(4) != 0, "tag_missed": stale and r(6) == 0}) return out def oracle(a): return {"storable": a["method"] in SAFE and not a["private"], "shareable": not a["private"], "fresh": not a["stale"], "repeatable": a["method"] in IDEMPOTENT, "redirectable": True} def message(a): return {"method": a["method"], "path": a["path"], "host": "station.example", "private_marker": a["private_marker"], "validator": a["stale"] and not a["tag_missed"]} def infer(i): k = {} if "method" in i: k["repeatable"] = i["method"] in IDEMPOTENT if "private_marker" in i: k["shareable"] = not i["private_marker"] if "method" in i: k["storable"] = i["method"] in SAFE and not i["private_marker"] if "validator" in i: k["fresh"] = not i["validator"] if "host" in i: k["redirectable"] = True return k def frame(batch, connections=1, compress=True): """Turns each message into a (bytes, literal-fields) pair. A name-value pair already seen on the same connection is sent as an index: it costs 1 byte, and an intermediary with no table cannot decode it.""" table, out = [set() for _ in range(connections)], [] for i, a in enumerate(batch): t, b, literal = table[i % connections], 0, set() for k, v in message(a).items(): if not compress or (k, v) not in t: b += len(k) + len(str(v)) + 4 literal.add(k) t.add((k, v)) else: b += 1 out.append((b, literal)) return out def measure(batch, frames, context): d = y = e = t = b = 0 for a, (byte_cost, literal) in zip(batch, frames): visible = set(FIELDS) if context else literal truth = oracle(a) given = infer({k: v for k, v in message(a).items() if k in visible}) missing = False for decision in DECISIONS: if decision not in given: e += 1 missing = True elif given[decision] == truth[decision]: d += 1 else: y += 1 t += 1 if missing else 0 b += byte_cost return d, y, e, t, b def blocking_units(batch, blocked, connections=1): """An exchange waits one unit behind every blocker ahead of it in the same queue.""" prior, total = [0] * connections, 0 for i, blk in enumerate(blocked): total += prior[i % connections] prior[i % connections] += 1 if blk else 0 return total batch = exchanges() loss_gen = generator(SEED + 1) SLOW = [a["path"] == "/summary" for a in batch] LOST = [loss_gen(5) == 0 for _ in batch] BOTH = [s or l for s, l in zip(SLOW, LOST)] print(f"exchanges {len(batch)} | decisions {len(batch) * len(DECISIONS)} | " f"assembled response {sum(SLOW)} | lost in transport {sum(LOST)}") print() print("framing regime correct wrong unavailable extra round bytes") for name, frames, context in ( ("text line", frame(batch, 1, False), True), ("binary, context kept", frame(batch), True), ("binary, no context", frame(batch), False)): d, y, e, t, b = measure(batch, frames, context) print(f" {name:23s} {d:5d} {y:7d} {e:12d} {t:13d} {b:6d}") print() print("framing regime app order transport order blocked exchanges") for name, app_order, transport_order in (("text line, single connection", True, True), ("binary, single transport stream", False, True), ("binary, separate transport streams", False, False)): blocked = BOTH if app_order and transport_order else LOST if transport_order else [False] * len(batch) print(f" {name:36s} {'yes' if app_order else 'no':>9s}" f" {'yes' if transport_order else 'no':>16s} {blocking_units(batch, blocked):18d}") print() print("parallel connections blocked exchanges literal fields bytes") for n in (1, 2, 6): c = frame(batch, n) print(f" {n:19d} {blocking_units(batch, BOTH, n):19d}" f" {sum(len(s) for _, s in c):15d} {sum(b for b, _ in c):6d}") ``` ``` exchanges 40 | decisions 200 | assembled response 9 | lost in transport 7 framing regime correct wrong unavailable extra round bytes text line 195 5 0 0 3861 binary, context kept 195 5 0 0 459 binary, no context 10 1 189 39 459 framing regime app order transport order blocked exchanges text line, single connection yes yes 251 binary, single transport stream no yes 83 binary, separate transport streams no no 0 parallel connections blocked exchanges literal fields bytes 1 251 15 459 2 121 29 693 6 36 74 1481 ``` #### The Gain: Same Decision, an Eighth of the Bytes The first table's first two rows should be read side by side. Text-line framing gives **195** correct, **5** wrong, **0** unavailable, and spends **3861** bytes. Binary framing that keeps context gives the **same** three numbers and spends **459** bytes. The decision table does not change; bytes drop **88.2%**. This is the shared setup's second claim read backwards. Adding a field did not buy a decision; now it turns out that **making a field cheaper does not cost one either.** As long as the intermediary can resolve a field, its byte cost is unrelated to the decision. Two hundred field occurrences pass through the forty exchanges, and only **15** are a distinct name–value pair; the remaining 185 repeat the same fifteen pairs. This is exactly where compression gains ground. The second table gives the second gain. Text-line framing on a single connection produces **251** blocking units in total: nine assembled responses and seven losses each hold up every exchange behind them. Once multiplexing removes application order, blocking drops to **83** — all of it now from the transport layer. Once the transport separates streams too, it is **0**. The order of the three rows matters: multiplexing does not zero out blocking, it **drops it to a third**; what zeros it out is the transport changing. The rest of the problem was sitting one layer below the one framing solved. #### The Loss: An Unresolvable Field Is Ignored The third row is this lesson's real finding. Bytes stay the same **459**, but the decision table collapses: **10** correct, **1** wrong, **189** unavailable, **39** extra round trips. The reason: across forty messages, only **15** fields are written literally; the remaining 185 are sent by index. An intermediary with no table cannot read those 185 fields, and it cannot draw a decision from a field it cannot read. **189** of the two hundred decisions move from the middle to the endpoints — more even than the 160 the envelope regime carried. The fields **sit there** in the message; they do not exist for the intermediary. The remaining eleven decisions come from the first exchange and the first few occurrences where the table gets built. The single wrong decision sits there too: the first exchange is `POST /register` and is personalized, but the server did not set the flag. The intermediary can read the field in that message, so it decides, and decides **wrong**. The pattern does not change — a readable field can be wrong; an unreadable one is only missing. The rule that follows: **compression is a visibility decision.** An intermediary present from the connection's start loses nothing; one that joins later, or forwards frames without decoding them, loses almost everything. Text-line framing had no such split, because every message was self-sufficient. #### Parallel Connections: Another Price for the Same Problem The third table measures the pre-multiplexing fix: opening more than one connection to the same host. Blocking genuinely drops — from **251** to **121** on two connections, to **36** on six. The price sits in the two columns to the right. Every connection has its own header table; as exchanges split across connections, the same name–value pairs get written literally again on each one. The count of literal fields rises from **15** to **29**, then to **74**; bytes rise from **459** to **693**, then to **1481**. Blocking dropping to a seventh on six connections is paid for by bytes tripling. This is where multiplexing's difference shows: on one connection, it **both** pushes blocking down to the transport layer **and** keeps the table in one piece. Parallel connections trade one cost for the other instead of removing either. #### Summary - The difference between versions is not in meaning but in framing: the same fields are written as a text line or as a binary frame with a stream identifier. - Binary framing that keeps context gives the **same** 195 correct and 5 wrong decisions for **459** bytes instead of **3861**; only **15** of the 200 field occurrences across forty exchanges are a distinct name–value pair. - Head-of-line blocking has two forms; multiplexing removes the application-layer one and drops blocking from **251** to **83**, and only the transport gaining stream awareness drops the transport-layer form to **0**. - An intermediary with no context reads only fields written literally: **189** decisions move to the endpoints, a debt of **39** round trips is created, and bytes do not change at all. - Parallel connections drop blocking to **36** but split the table; literal fields rise to **74**, bytes to **1481**. #### Next Step Framing determines whether the intermediary can read a field. But even when it can read one, there is a decision that comes not from the field itself but from **comparing two values**: does the copy on hand say the same thing as the message that just arrived? The next lesson measures that comparison. Which field tells the intermediary whether its stored copy is still valid, what that field's resolution is, and what the intermediary says when the resolution falls short. ### HTTP Caching Source: https://academia.sh/en/courses/application-protocols/http-caching Freshness and validation are separate questions, and the validator has a resolution: it catches 9 of the 12 stale copies across forty exchanges, misses the tag change on 3, and the intermediary calls them fresh — three of the five wrong decisions in the open regime come from here. The previous lesson measured how framing determines which field the intermediary can read. But being able to read a field is not always enough. Four of the five decisions look at a single field: the method says retryability, the flag says shareability. The fifth is different. **Freshness** does not come from one field's value; it comes from **comparing** the copy on hand against the message that just arrived. Comparison brings a new problem. Any mechanism comparing two values has a **resolution**: a difference smaller than what it can distinguish is no difference to it. This is what this lesson measures — how often the intermediary notices its stored copy has gone stale, and what it says when it does not. #### Freshness and Validation: Two Separate Questions There are two separate questions for a copy in the intermediary's cache. **The freshness question:** can I give out this copy **without asking**? The answer is the **freshness lifetime** the server declares. Within that lifetime, the intermediary hands out the copy directly, never returning to the server. **The validation question:** once the lifetime has run out, is the copy still valid? The answer comes from the **validator**: a tag the server attaches to the representation. The intermediary compares its held tag against the server's; a match means the copy is valid. Confusing the two loses the measure. Freshness is a **duration** decision and spends no round trip. Validation is an **identity** decision and gives the truth. Caching's whole design rests on the trade-off between the two: stretch the duration and round trips fall, but the odds of handing out a stale copy rise. A third question comes before these two, established in the previous two lessons: is this response **storable**, and if stored, is it **shareable**? The intermediary stores a copy only if the method is safe, and hands it to someone else only if there is no personalization flag. Freshness follows these two decisions; there is no freshness for a copy never stored. The measurement preserves this order: all three regimes give the same `storable` and `shareable` decisions, differing only in freshness. What this course measures is not the directive list or the body caching saves; those belong to the **Caching, Queues and Asynchronous Processing** course. The only thing measured here is the **validator's resolution** — which changes the tag sees, and which it does not. #### The Validator's Resolution HTTP defines two kinds of validator, and their resolutions are not the same. The **timestamp** validator carries the representation's last moment of change. Its resolution is the time unit the tag is written in: a representation changing twice within the same second carries the same tag after both changes, and the intermediary sees no difference. The **entity tag** validator is a string derived from the representation's content and is not tied to a time unit. It has a **strong** and a **weak** form. A strong tag carries byte-level distinction. A weak tag deliberately disregards changes considered semantically unimportant — a validator whose **resolution is lowered on purpose.** ```text # teaching dump, not run # validators the server carries in the response HTTP/1.1 200 OK Last-Modified: Sun, 09 Aug 2026 11:04:07 GMT ETag: "measurement-7a" Cache-Control: max-age=60 Age: 12 # the conditional request the intermediary builds to test its copy GET /measurement/north HTTP/1.1 Host: station.example If-None-Match: "measurement-7a" If-Modified-Since: Sun, 09 Aug 2026 11:04:07 GMT # if the tag is unchanged HTTP/1.1 304 Not Modified ETag: "measurement-7a" ``` `Cache-Control` declares the freshness lifetime, `Age` declares how long the copy has spent in cache; together they answer the freshness question. `If-None-Match` and `If-Modified-Since` build the **conditional request**, and `304` is the status code declaring the tag stayed the same. The measurement reduces all these fields to one declaration: does the tag in the message differ from the one the intermediary holds? `Age` has a separate function: the intermediary declares its stored copy's age through this field, so the next intermediary in the chain can do the same accounting. With more than one intermediary, age accumulates as it travels, and the freshness lifetime is **a single budget** for the whole chain; each stop spends its own share. The tag itself is nothing more than what is written in the message. **Nothing verifies the validator:** if the tag has not changed, the intermediary believes the representation has not changed, because it has no other basis to go on. - **HA55** — The same forty exchanges and oracle are used; the oracle knows whether the intermediary's stored copy is actually stale. - **HA56** — The intermediary sits on the path and sees passing responses; it compares its held tag against the one arriving in the message. This comparison **needs no extra round trip**, since the message already passes in front of it. - **HA57** — The tag has a resolution: if a change is smaller than what the tag can distinguish, the tag stays the same and the intermediary cannot see the change. - **HA58** — Three regimes are measured. In `duration only`, the message declares a freshness lifetime and no validator. In `tag`, the validator is declared. In `always ask`, neither is declared. - **HA59** — The freshness lifetime does not run out within the measurement window; in `duration only`, the intermediary counts every copy as fresh. - **HA60** — An intermediary that cannot make the freshness decision sends a conditional request; the server knows the truth, so catching after the round trip is complete. - **HA61** — The byte count is a count of the message's fields; the body is not counted. #### The Measurement ```python """HTTP caching: freshness, validation, and the validator's resolution. Part 1 - how many stale copies the validator catches. Part 2 - three regimes: duration only, tag, always ask. """ SEED = 20260809 SAFE = {"GET", "HEAD"} IDEMPOTENT = {"GET", "HEAD", "PUT", "DELETE"} METHOD_POOL = ["GET", "GET", "GET", "HEAD", "POST", "PUT", "DELETE"] PATH_POOL = ["/measurement/north", "/measurement/slope", "/summary", "/register", "/session"] DECISIONS = ("storable", "shareable", "fresh", "repeatable", "redirectable") def generator(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def exchanges(count=40): r, out = generator(SEED), [] for i in range(count): method, path = METHOD_POOL[r(7)], PATH_POOL[r(5)] private = path == "/session" or r(5) == 0 stale = r(3) == 0 out.append({"num": i + 1, "method": method, "path": path, "private": private, "stale": stale, "private_marker": private and r(4) != 0, "tag_missed": stale and r(6) == 0}) return out def oracle(a): return {"storable": a["method"] in SAFE and not a["private"], "shareable": not a["private"], "fresh": not a["stale"], "repeatable": a["method"] in IDEMPOTENT, "redirectable": True} def message(a, visible): full = {"method": a["method"], "path": a["path"], "host": "station.example", "private_marker": a["private_marker"], "validator": a["stale"] and not a["tag_missed"], "freshness_window": True} return {k: v for k, v in full.items() if k in visible} def byte_count(i): return sum(len(k) + len(str(v)) + 4 for k, v in i.items()) def infer(i): k = {} if "method" in i: k["repeatable"] = i["method"] in IDEMPOTENT if "private_marker" in i: k["shareable"] = not i["private_marker"] if "method" in i: k["storable"] = i["method"] in SAFE and not i["private_marker"] if "validator" in i: k["fresh"] = not i["validator"] elif "freshness_window" in i: k["fresh"] = i["freshness_window"] if "host" in i: k["redirectable"] = True return k COMMON = ("method", "path", "host", "private_marker") VISIBLE = {"duration only": COMMON + ("freshness_window",), "tag": COMMON + ("validator",), "always ask": COMMON} batch = exchanges() STALE = [a for a in batch if a["stale"]] MISSED = [a for a in STALE if a["tag_missed"]] print(f"exchanges {len(batch)} | decisions {len(batch) * len(DECISIONS)} | " f"stale copy {len(STALE)} | missed tag change {len(MISSED)}") print("exchanges the tag missed:", [(a["num"], a["path"]) for a in MISSED]) print() print(f" {'regime':18s}{'caught':>8s}{'missed':>8s}{'resolution':>12s}" f"{'rounds asked':>14s}{'after round trip':>18s}") for name in VISIBLE: caught = sum(1 for a in STALE if infer(message(a, VISIBLE[name])).get("fresh") is False) asked = sum(1 for a in batch if "fresh" not in infer(message(a, VISIBLE[name]))) print(f" {name:18s}{caught:>8d}{len(STALE) - caught:>8d}" f"{caught / len(STALE):>12.4f}{asked:>14d}" f"{len(STALE) if asked else caught:>18d}") print() print(f" {'regime':18s}{'decision':9s}{'correct':>8s}{'wrong':>8s}" f"{'unavailable':>13s}{'extra round':>14s}{'bytes':>7s}") for name in VISIBLE: fd = fy = fe = d = y = e = t = b = 0 for a in batch: truth, given = oracle(a), infer(message(a, VISIBLE[name])) missing = False for decision in DECISIONS: if decision not in given: e += 1 missing = True fe += decision == "fresh" elif given[decision] == truth[decision]: d += 1 fd += decision == "fresh" else: y += 1 fy += decision == "fresh" t += 1 if missing else 0 b += byte_count(message(a, VISIBLE[name])) print(f" {name:18s}{'freshness':9s}{fd:>8d}{fy:>8d}{fe:>13d}") print(f" {name:18s}{'all':9s}{d:>8d}{y:>8d}{e:>13d}{t:>14d}{b:>7d}") ``` ``` exchanges 40 | decisions 200 | stale copy 12 | missed tag change 3 exchanges the tag missed: [(24, '/summary'), (25, '/session'), (37, '/measurement/slope')] regime caught missed resolution rounds asked after round trip duration only 0 12 0.0000 0 0 tag 9 3 0.7500 0 9 always ask 0 12 0.0000 40 12 regime decision correct wrong unavailable extra round bytes duration only freshness 28 12 0 duration only all 186 14 0 0 4110 tag freshness 37 3 0 tag all 195 5 0 0 3861 always ask freshness 0 0 40 always ask all 158 2 40 40 3150 ``` #### Resolution: Three-Quarters The upper table's middle row is this lesson's number. Across forty exchanges, the intermediary's stored copy is stale **12** times. The validator sees the tag changed in **9** of those and counts the copy stale. In **3**, the tag has not changed; the intermediary sees the two tags as identical and calls the copy **fresh**. Resolution **0.7500**. The three missed exchanges have different paths: `/summary`, `/session`, and `/measurement/slope`. The miss depends not on a path or a method but on the change being smaller than what the tag can distinguish. This follows from the definition of resolution and is not a fixable bug; it is a range narrowed only by changing the tag. None of these three misses show any sign: the intermediary followed the rule, compared the tags correctly, and handed out the copy for a valid reason. Only the oracle knows the decision was wrong, and it exists only because we generated the setup ourselves. In a real deployment, no such oracle exists: **a stale copy looks identical to a fresh one.** One more way to drop a stored copy costs no round trip: seeing a `PUT` or `DELETE` on its stored path, the intermediary infers that copy is no longer valid and drops it — a decision drawn from the message, independent of the tag's resolution. Its limit: the intermediary sees only the request that **passes in front of it**. A request changing the same representation through another path does not, and the copy silently goes stale — the same outcome as the three exchanges the tag missed. The upper table's first row gives the case with no validator. The intermediary only looks at the freshness lifetime, which never runs out: it catches **none** of the **12** stale copies. Resolution **0.0000**. Duration is not even a resolution measure, since it never tests for change — it only places a bet. The last row does the opposite: the intermediary asks on every exchange, **40** round trips, and catching is complete, **12/12**. But forty questions were asked to catch those twelve copies; twenty-eight were spent on copies already fresh. #### The Decision Cost of Three Regimes The lower table gives the same three regimes across the full decision set. In **`duration only`**, the freshness decision is **28** correct, **12** wrong; the full set is **186**/**14**/**0**. The wrong count peaks here: without asking once, without leaving a single decision unavailable, the intermediary is wrong fourteen times. Bytes: **4110**, the highest of the three regimes. In **`tag`**, freshness is **37** correct, **3** wrong; the full set is **195**/**5**/**0** and **0** round trips. This is exactly the shared setup's open regime, and **three of its five wrong decisions come from this row**. The remaining two come from the personalization flag the server forgot to set and have nothing to do with freshness. In **`always ask`**, freshness is never decided on its own: **40** unavailable. The full set is **158**/**2**/**40** and **40** round trips. Wrong drops from **5** to **2**; the remaining two are again from the personalization flag — asking does not fix that, since the question asked concerns freshness. The three rows give the course's axis once more. **An unavailable decision is a round-trip debt:** forty round trips for forty decisions. **A decision that is made can still be wrong:** the tag regime takes all forty freshness decisions with zero round trips and is wrong on three. The duration regime also spends zero round trips but is wrong on twelve — the same cost, four times the wrong count. #### What Ranking Depends On There is no single "best" among the three regimes; ranking changes with what is expensive. If round trips are expensive, `tag` is chosen: zero round trips, three wrong. Three wrong decisions, in a set of two hundred, is a share of **0.0150**; since the smallest measurable difference here is **1/200 = 0.0050**, three wrong decisions sits inside the band and is defensible. If handing out a stale copy is unacceptable, `always ask` is chosen and forty round trips are paid. `duration only` is defensible only for representations not expected to change; with a stale rate of **12/40** here, it is not defensible. Raising the tag's resolution is a fourth path, absent from the table since the measurement takes the tag as given. Still, the direction is clear: an entity tag over a timestamp, a strong tag over a weak one, narrow resolution rather than widen it. Each narrowing reduces the three misses, and none adds a round trip. One last observation: `always ask`'s bytes are **3150**, `tag`'s are **3861**. Asking shrinks the message, since a field not carried takes no bytes — but the smaller message is nothing next to the forty round trips owed. **Bytes are cheap, round trips are expensive** — the course's measurement axis has said exactly this all along. #### Summary - Freshness and validation are separate questions: freshness answers whether the copy can be given out **without asking**; validation answers whether it is still valid. - The validator has a resolution; a timestamp is limited by its time unit, an entity tag by content, and a weak tag deliberately lowers resolution. - **9** of the **12** stale copies across forty exchanges are caught, **3** miss the tag change and the intermediary calls them **fresh**; resolution **0.7500**. - **3** of the open regime's **5** wrong decisions come from these three copies; the remaining two come from the personalization flag, unrelated to freshness. - Three regimes: `duration only` **186**/**14**/**0** and 0 round trips, `tag` **195**/**5**/**0** and 0 round trips, `always ask` **158**/**2**/**40** and 40 round trips. Asking lowers wrong decisions and creates a round-trip debt. #### Next Step Throughout this topic, the intermediary made its decisions for the same reason every time: **it could read what was written in the message.** It read the method and decided retryability, read the flag and decided shareability, read the tag and decided freshness. Even where it was wrong, it could still read; the error came not from failing to read but from what it read being incomplete. The next topic removes this condition. **What if the message cannot be read?** If the two endpoints turn the message between them into something only they can understand, what is left in the intermediary's hands? The question connects to the course's third claim: the endpoints are not the only ones making decisions, and closing off the message moves decisions from the middle out to the endpoints. The next lesson takes up how that closing off is built, and who pays what for it. ## Secure Transport The roles of symmetric and asymmetric encryption, verifying the certificate chain, the handshake's round-trip budget, and the certificate's life cycle. ### Symmetric and Asymmetric Encryption Source: https://academia.sh/en/courses/application-protocols/symmetric-and-asymmetric-encryption The roles of the two key models: asymmetric in setup, symmetric in transport. The round-trip cost of setup count per exchange, and the visibility the intermediary loses. Every decision counted so far rested on a single assumption: the intermediary can read the message. Because it read the method, the path, the private marker, and the validator, it could take the `repeatable`, `cacheable`, and `fresh` decisions with no extra round trip. This assumption has a cost, and the cost is silent: the layer that can read the message is not only the intermediary in the scenario, it is **every** layer on the path. This lesson's question is what mechanism makes the message unreadable, and what it costs. The measure does not change — we still count round trips and decisions. What changes is that what is measured is no longer a field but a **role**: which key model runs at which step, and how many rounds that step holds. #### Two Key Models **Symmetric encryption** uses a single secret. The same secret sits on both sides; the sender closes the body with it, the receiver opens it with the same secret. Its cost scales with the body's length and carries no round cost — the secret is already on both sides. It has one question: **how did the secret reach both sides?** The question looks small but alone it makes the symmetric model unfit for a network protocol. If the secret must be handed over in advance, every pair of endpoints that wants to talk needs its own secret, and the number of secrets needed grows not with the number of endpoints but with the number of **endpoint pairs**. Worse, two endpoints that have never met can never talk. By definition, endpoints on a network do not know each other in advance; this is the first problem transport security has to solve. **Asymmetric encryption** uses a pair. The **public key** is distributed, the **private key** is never sent. The pair works because of this: when one side combines its own private value with the other side's public value, and the other side combines its own private value with the first side's public value, the **same number** comes out. This number is never carried on the wire; it is produced separately at both ends. This step is called **key exchange**, and the number that comes out is called the **shared secret**. The Identity, Access and Cryptography course established the security properties of these two models: under what assumption something counts as unbreakable, what a signature proves, what key length means. That theory is not repeated here. The difference sits in one sentence: there, a **security property** was measured; here, **rounds and decisions** are measured. #### Order of Roles The two models are not rivals; they run in sequence. ```text setup — asymmetric roles 1. client -> server : the client's public value 2. server -> client : the server's public value 3. both sides reach the shared secret on their own (nothing is carried on the wire) transport — symmetric role 4. client -> server : body closed with the shared secret 5. server -> client : body closed with the shared secret 6. 4-5 repeat as many times as needed; 1-3 do not repeat ``` This listing is not a specification summary; it is the order of the roles, and it is not run. The line to notice is the sixth: **setup once, transport many times.** #### Mechanism Modeled Within the Lesson The mechanism below is **modeled within the lesson.** It is not a real key exchange, not a real cipher, and carries no security property: the modulus is shrunk, the private numbers are drawn from a narrow range, the symmetric transform is a toy. The only thing modeled is **how the sharing happens** and **which work happens at which step**. ```python SEED = 20260809 PRIME = 2147483647 # modulus shrunk for the lesson BASE = 5 def generator(seed: int): d = seed % 2147483646 + 1 def r(n: int) -> int: nonlocal d d = (d * 48271) % PRIME return d % n return r def public_value(private: int) -> int: return pow(BASE, private, PRIME) def shared_secret(own_private: int, peer_public: int) -> int: return pow(peer_public, own_private, PRIME) def symmetric(data: bytes, secret: int) -> bytes: r = generator(secret) return bytes(b ^ r(251) for b in data) r = generator(SEED) messages = [bytes(r(251) for _ in range(180 + r(441))) for _ in range(40)] client_private, server_private = 1 + r(9973), 1 + r(9973) client_secret = shared_secret(client_private, public_value(server_private)) server_secret = shared_secret(server_private, public_value(client_private)) carried = [symmetric(m, client_secret) for m in messages] decrypted = [symmetric(c, server_secret) for c in carried] print(f"shared secret matches: {client_secret == server_secret}") print(f"messages {len(messages)}, body {sum(len(m) for m in messages)} bytes") print(f"carried {sum(len(c) for c in carried)} bytes, decrypted identical: {decrypted == messages}") ``` ``` shared secret matches: True messages 40, body 15767 bytes carried 15767 bytes, decrypted identical: True ``` Both sides used only their own private value and the other side's public value; the private values were never sent, and both sides arrived at the same number. The body was folded with a sequence derived from that number; applying the same sequence a second time brought the body back. The carried byte count staying equal to the body is no coincidence either: the symmetric role does not **lengthen** the data, it only makes it unreadable. In a real transport every part of this mechanism is heavier: the modulus is chosen far larger, the private numbers are drawn from a far wider range, and the symmetric transform not only closes the body but also reports whether it was altered. The justification for these choices belongs to the Identity, Access and Cryptography course. The only thing taken from the model here is the **division of labor**. #### Setup Once, Transport Many Times If all forty exchanges run through the same mechanism, the only thing that changes is **how many times setup runs**. The measurement below counts this across three regimes: one setup per message, setup split across eight connections, one persistent connection. Assumptions of the measurement: - **GT1** — The encryption itself is **modeled within the lesson**: the modulus is shrunk, the private numbers are drawn from a narrow range, and the symmetric transform is a fold. No real algorithm runs and the mechanism carries no security property. - **GT2** — The body is produced from a single seed: forty messages, each between 180 and 620 bytes. The same body is used in all three regimes. - **GT3** — One setup counts as one round trip; at each setup both sides draw a **new** private value and two public values are produced. - **GT4** — Three regimes are compared: setup per message, eight connections, one persistent connection. One setup runs per connection, and the secret from that setup covers the whole connection's exchanges. - **GT5** — The smallest round difference measurable across a set of forty exchanges is one setup, i.e. 1/40 = 0.025; a gain smaller than this cannot be defended with this set. - **GT6** — Role cost is measured not by time but by **operation count**: exponentiation is opened into square-and-multiply and modular multiplications are counted, folds are counted for the symmetric transform. What is counted in this lesson is rounds; the intermediary's decision table does not enter the measurement. ```python SEED = 20260809 PRIME = 2147483647 BASE = 5 COUNTER = {"public_value": 0} def generator(seed: int): d = seed % 2147483646 + 1 def r(n: int) -> int: nonlocal d d = (d * 48271) % PRIME return d % n return r def public_value(private: int) -> int: COUNTER["public_value"] += 1 return pow(BASE, private, PRIME) def setup(rk) -> int: client_private, server_private = 1 + rk(9973), 1 + rk(9973) public_value(client_private) # value sent to the server return pow(public_value(server_private), client_private, PRIME) def symmetric(data: bytes, secret: int) -> bytes: r = generator(secret) return bytes(b ^ r(251) for b in data) r = generator(SEED) messages = [bytes(r(251) for _ in range(180 + r(441))) for _ in range(40)] print(f"{'regime':<27s} {'setups':>8s} {'public values':>13s} {'symmetric bytes':>16s} " f"{'round trips per exchange':>25s} {'exchanges per secret':>21s}") for name, connections in (("setup per message", 40), ("eight connections", 8), ("one persistent connection", 1)): COUNTER["public_value"] = 0 rk, total_bytes = generator(SEED + 1), 0 secrets = [setup(rk) for _ in range(connections)] for idx, message in enumerate(messages): total_bytes += len(symmetric(message, secrets[idx % connections])) print(f"{name:<27s} {connections:8d} {COUNTER['public_value']:13d} {total_bytes:16d} " f"{connections / len(messages):25.3f} {len(messages) / connections:21.1f}") ``` ``` regime setups public values symmetric bytes round trips per exchange exchanges per secret setup per message 40 80 15767 1.000 1.0 eight connections 8 16 15767 0.200 5.0 one persistent connection 1 2 15767 0.025 40.0 ``` Across the three rows the **symmetric byte count does not change**: the body is the same in every regime, 15767 bytes. The only thing that changes is the setup count and the round it produces. In the regime that sets up on every message, each exchange carries **1.000** extra round trip — every message pays for its own setup. In eight connections this becomes **0.200**, in one persistent connection **0.025**. The forty-fold difference comes from a single design decision. The asymmetric role's cost does not scale with the body, it scales with the **setup count**; the symmetric role's cost does not scale with the setup count, it scales with the **body**. This is the entire reason the two models are used together: the expensive one runs rarely, the cheap one runs often. This set's resolution is also read here. The smallest round difference measurable across a set of forty exchanges is **one setup**, i.e. 1/40 = 0.025. A gain smaller than this cannot be claimed. The last column shows it is not free. As setups become rarer, **the number of exchanges tied to one secret** grows: at setup per message one secret covers one exchange, at eight connections five, at one persistent connection all forty at once. The round gain and isolation pull the same lever in opposite directions. This is also why it matters that both sides draw a **new** private value on every repetition of setup: a new private value re-bounds the scope to a single connection. Reusing the private value — using the same one across every setup — costs nothing in the table but silently pushes the last column to forty. #### Why Transport Is Symmetric The order of roles is not a habit, it is the outcome of a calculation. In the modeled mechanism, the cost of the two jobs can be counted in the same unit: the asymmetric role does **modular multiplication**, the symmetric role does **folding**. Opening the exponentiation into square-and-multiply and counting every multiplication shows where the order comes from. The same body is carried by both roles and both round-trip back open. ```python SEED = 20260809 PRIME = 2147483647 BASE = 5 PUBLIC_EXP = 1223 PRIVATE_EXP = pow(PUBLIC_EXP, -1, PRIME - 1) def generator(seed: int): d = seed % 2147483646 + 1 def r(n: int) -> int: nonlocal d d = (d * 48271) % PRIME return d % n return r def power(base: int, exponent: int, counter: list[int]) -> int: """Square-and-multiply: every modular multiplication is added to the counter.""" result, t = 1, base % PRIME while exponent > 0: if exponent % 2 == 1: result = (result * t) % PRIME counter[0] += 1 t = (t * t) % PRIME counter[0] += 1 exponent //= 2 return result def fold(data: bytes, secret: int, counter: list[int]) -> bytes: r = generator(secret) counter[0] += len(data) return bytes(b ^ r(251) for b in data) r = generator(SEED) body = b"".join(bytes(r(251) for _ in range(180 + r(441))) for _ in range(40)) setup = [0] client_private, server_private = 1 + r(9973), 1 + r(9973) server_public = power(BASE, server_private, setup) power(BASE, client_private, setup) secret = power(server_public, client_private, setup) asymmetric = [0] closed = [power(b, PUBLIC_EXP, asymmetric) for b in body] opened = bytes(power(x, PRIVATE_EXP, asymmetric) for x in closed) symmetric = [0] s_opened = fold(fold(body, secret, symmetric), secret, symmetric) print(f"{'work':<34s} {'modular multiplications':>24s} {'folds':>9s}") print(f"{'setup (three exponentiations)':<34s} {setup[0]:24d} {0:9d}") print(f"{'carrying the body asymmetrically':<34s} {asymmetric[0]:24d} {0:9d}") print(f"{'carrying the body symmetrically':<34s} {0:24d} {symmetric[0]:9d}") print() print(f"body {len(body)} bytes, asymmetric round trip: {opened == body}, " f"symmetric round trip: {s_opened == body}") print(f"asymmetric transport is {asymmetric[0] / setup[0]:.0f}x the setup") ``` ``` work modular multiplications folds setup (three exponentiations) 54 0 carrying the body asymmetrically 930253 0 carrying the body symmetrically 0 31534 body 15767 bytes, asymmetric round trip: True, symmetric round trip: True asymmetric transport is 17227x the setup ``` Setup holds three exponentiations and does **54** modular multiplications total. However large the body grows, this number does not change, because the asymmetric role touches not the body but a **fixed-size number**. Carrying the same body end to end with the asymmetric role alone costs **930253** modular multiplications: **17227 times** the setup. The symmetric role finishes the same work in **31534** folds and does not perform a single modular multiplication. What matters is not the size of the numbers but **what they depend on**. Setup's cost is independent of the body; asymmetric transport's cost grows linearly with the body and runs every byte through an expensive operation; symmetric transport's cost also grows linearly with the body but the work it does per byte is cheap. The order of the roles comes out of comparing these three curves: the expensive one runs where it can stay constant, the cheap one runs where it can grow. One more detail remains. The shared secret coming out of setup is not applied directly to the body; a separate **session key** derived from it is used, and usually a separate key is derived for each direction. Derivation is a local operation, it sends no message and adds no round. What it gains is the separation of the two directions and the two sessions from each other. In terms of the round budget, the result is this: even as the number of derived keys grows, **the setup's round stays fixed.** #### What the Intermediary Sees and Does Not Once the body is closed, the intermediary cannot take a decision from the body. The transform this lesson modeled closed only the body; the **transport envelope**, the subject of the next two lessons, also folds in the header fields and narrows the set the intermediary can read even further. What that narrowing does to the decision table is counted in the handshake lesson. What is counted here is only the round. In a closed transport there are things the intermediary still sees, and each has a narrowing: - **Host name.** The intermediary needs a name to route the message to the right server. *Narrowing:* if the name is also folded into the envelope, the intermediary cannot route; this is not a loss but a trade — the decision is taken from the intermediary, given to the endpoint. - **Message length.** Since the symmetric role does not lengthen the data, the bytes carried are the body's bytes. *Narrowing:* if padding is added to the body, length becomes a coarse band and its distinctiveness drops. - **Round pattern.** How many messages go, at what interval. *Narrowing:* if the exchanges are merged into a single persistent connection, the pattern is buried inside one connection. - **Destination address and port.** The transport layer's headers stay outside the closed body; the intermediary must read them to be able to forward the message. *Narrowing:* if many endpoints are gathered behind the same address, the address points to a set rather than a single endpoint and its distinctiveness drops. There is also a choice that must be read as a **configuration flaw**: skipping the setup round entirely and binding the shared secret to a fixed, pre-shared value. The gain is obvious — even the 0.025 row disappears from the table. What is wrong is this: every session binds to the same secret, sessions cannot be separated from each other, and a single leak of the secret also covers past exchanges. What this lesson states is what is wrong; how this is used is not this course's subject. #### Summary - The symmetric model uses a single secret and its cost scales with the body; the asymmetric model uses a pair and its cost scales with the setup count. - Key exchange is two sides arriving at the same shared secret without sending their private values; the shared secret is never carried on the wire. - In the mechanism modeled within the lesson, both sides arrived at the same number and the 15767-byte body round-tripped through the transform identically; the mechanism carries no security property. - Across forty exchanges, extra round trip per exchange comes out to 1.000 at setup per message, 0.200 at eight connections, 0.025 at one persistent connection — the symmetric byte count is the same in all three. - In closed transport, host name, message length, and round pattern each have a narrowing; binding the shared secret to a fixed value is a configuration flaw. #### Next Step Setup produced a number, and both sides agreed on that number. But what was agreed is the **secret, not the identity**: the shared secret says nothing about who the other end is. The side that sent the public value could be the measurement station, or it could be some other endpoint using that name. The only thing in the client's hand is still what the message says — and nothing verifies what the message says. The next lesson measures how a name binds to a public key, and how many times the client asks a question while verifying that bond. ### Certificates and the Trust Chain Source: https://academia.sh/en/courses/application-protocols/certificates-and-the-trust-chain A name binding to a public key, the chain verifying locally all the way to the trust anchor, and this decision never touching the round-trip budget. The previous lesson showed that setup produces a shared secret and counted how many rounds that secret costs. What was produced was named a secret; it was **not identity.** The client took the other side's public value, arrived at a shared number with it, and closed the body with that number — but nothing said the other end was the measurement station. The side that sent the public value could be `station.example`, or some other endpoint using that name. This lesson's question is how a **name** binds to a **public key**, and how many times the client asks a question while verifying that bond. Its second half is the course's axis: chain verification is **a decision taken with no extra round trip**, true even as the chain grows longer. #### What the Certificate Declares A certificate is a statement that binds a name to a public key and puts someone else's signature under that bond. Its fields say: ```text certificate subject : station.example <- whose name public key : <- which key speaks with that name issuer : intermediate-1.test <- who approved the bond validity : start .. end <- the time range the bond holds in signature : produced with the issuer's private key ``` This listing is not run; it shows what the fields declare. The body that issues the certificate is called a **certificate authority**. The intermediary's position is familiar here: it is still looking at what the message says. The difference is one field — here, something **verifies** what the message says: the signature. When the rest of the course says "the intermediary believes what it is told," it means the absence of this verification. #### What the Signature Proves The signature mechanism below is **modeled within the lesson.** It is not a real signature scheme and carries no security property: the modulus is shrunk, the digest function is a toy, and in this model the private key can be recovered from the public key. The security justification for real schemes belongs to the Identity, Access and Cryptography course. The only thing taken from the model here is **which fields the signature stands on.** ```python from math import gcd SEED = 20260809 PRIME = 2147483647 # modulus shrunk for the lesson def generator(seed: int): d = seed % 2147483646 + 1 def r(n: int) -> int: nonlocal d d = (d * 48271) % PRIME return d % n return r def digest(text: str) -> int: h = 2166136261 for b in text.encode(): h = (h * 16777619 + b) % PRIME return h def key_pair(r) -> tuple[int, int]: while True: private = 3 + r(PRIME - 4) if gcd(private, PRIME - 1) == 1: return private, pow(private, -1, PRIME - 1) def sign(subject: str, public: int, issuer: str, issuer_private: int, valid: bool) -> dict: s = {"subject": subject, "public": public, "issuer": issuer, "valid": valid} s["signature"] = pow(digest(f"{subject}|{public}|{issuer}|{valid}"), issuer_private, PRIME) return s def signature_valid(s: dict, issuer_public: int) -> bool: return pow(s["signature"], issuer_public, PRIME) == digest( f"{s['subject']}|{s['public']}|{s['issuer']}|{s['valid']}") r = generator(SEED) anchor_private, anchor_public = key_pair(r) leaf_private, leaf_public = key_pair(r) leaf = sign("station.example", leaf_public, "anchor.test", anchor_private, True) print(f"subject {leaf['subject']}, issuer {leaf['issuer']}") print(f"signature valid: {signature_valid(leaf, anchor_public)}") print(f"copy with altered subject: {signature_valid(dict(leaf, subject='other.test'), anchor_public)}") print(f"copy with altered key: {signature_valid(dict(leaf, public=7), anchor_public)}") ``` ``` subject station.example, issuer anchor.test signature valid: True copy with altered subject: False copy with altered key: False ``` The signature stands **on** the subject, the public key, the issuer, and the validity field; if any field changes, verification falls. In a real X.509 certificate the signature covers the **whole** to-be-signed portion of the document — the serial number and extensions such as the subject alternative name field discussed below included; the four fields in the model stand as one example of that whole. The result is subtle but important: the signature does not make the certificate unchangeable, it **makes a change show.** The signature stands not directly on the document but on the document's **digest**. This fixes the length of what is signed and makes the signing work independent of the document's size — the same division of labor as the previous lesson, the asymmetric role again touching a fixed-size number. In return a new condition is born: two different documents must not fall onto the same digest. Which assumption satisfies this condition belongs to the Identity, Access and Cryptography course. What the signature does not prove must also be listed. It shows the issuer approved this bond; it does not show the issuer was **authorized** to approve it, that the subject still exists at that moment, or that the subject's private key has not fallen into someone else's hands. All of these happen **outside** the certificate and cannot be written into a signed document afterward. #### Chain and Trust Anchor Verifying the signature needs the issuer's public key. What says that key belongs to the issuer? Another certificate. This question repeats itself and the **certificate chain** is born: at the bottom the **leaf certificate** carrying the subject, above it one or more **intermediate certificates**, at the top the point where the chain stops. The chain has to stop somewhere, and where it stops is decided not by a signature but by a **placement**. The certificate already in the client's hand, verified against nothing else, is called the **trust anchor**. The anchor can be **self-signed**, and usually is, but what makes it trustworthy is not its own signature — its own signature says nothing — but that **it is in the client's hand.** This distinction ties into the course's axis. The anchor sits with the client, not at the network's edge. So even for the chain's topmost rung, no one is asked a question. The answer to who carries the chain also decides the round budget. The **server sends** the leaf and intermediate certificates; the **client holds** the anchor. The client needs no separate request to collect the chain, because it already arrives inside the presentation. The server sending incomplete intermediate certificates, on the other hand, does not push the client to spend a round — it pushes straight to rejection: if the pieces in its hand do not reach the anchor, the client decides. #### Matching the Name Even if the chain is consistent, the decision depends on one more match: the name the client wants to reach must satisfy one of the names the certificate declares. X.509 certificates carry these names in the subject alternative name field; a single certificate can declare more than one name, and a wildcard form (like `*.station.example`) satisfies a single level. Two properties of the match tie into the course's axis. First, the match is also **local**: the client compares the name in its hand against the list in its hand, asking no one. Second and more subtle, the two ends of the comparison must come from **separate sources**: the requested name from the client's own request, the declared name from the certificate. If the name is read from the certificate and compared back against the same certificate, the check always passes and measures nothing — its meaning lies in the independence of its two ends. #### Chain Verification Needs No Round Trip Forty certificate submissions are measured. Chain lengths vary from one to five; submissions may have a rung with a broken signature, a chain that does not reach the client's anchors, a leaf subject that does not satisfy the requested name, or a closed validity field. The oracle — the truth known because we produced the scenario — states for each submission whether acceptance was correct. Each submission carries one decision, so the set is 40 decisions and its resolution is 1/40 = 0.025. Assumptions of the measurement: - **GT7** — The signature scheme is **modeled within the lesson**: the modulus is shrunk, the digest function is a toy, and in this model the private key can be recovered from the public key. What is measured is not the strength of the signature but **where** verification is done. - **GT8** — Forty submissions are produced from a single seed; chain length is drawn between one and five and each submission is given at most one flaw: broken signature, a chain that does not reach the anchor, a subject that does not satisfy the requested name, a closed validity field, or revocation. - **GT9** — The oracle is known because we produced the scenario: a submission's acceptance is correct only if it has no flaw. **The revocation flaw is not written into the chain** — those chains are consistent start to finish and verification cannot see revocation. - **GT10** — The server sends the leaf and intermediate certificates, the client holds the trust anchor; collecting the chain needs no separate request. The anchor set is singular. - **GT11** — Three regimes are compared: walking with a local anchor, asking each rung's issuer (one round trip per rung), never verifying at all. The walk **exits early**: it stops at a broken rung, so the signature-check count does not equal the rung count. - **GT12** — The decision measured belongs not to the intermediary but to the **client**; the intermediary does not enter this table. What is measured is not a duration but a verification step and a round trip. ```python from math import gcd SEED = 20260809 PRIME = 2147483647 FLAWS = ("none", "none", "none", "none", "none", "none", "none", "signature", "anchor", "name", "validity", "revoked", "revoked") def generator(seed: int): d = seed % 2147483646 + 1 def r(n: int) -> int: nonlocal d d = (d * 48271) % PRIME return d % n return r def digest(text: str) -> int: h = 2166136261 for b in text.encode(): h = (h * 16777619 + b) % PRIME return h def key_pair(r) -> tuple[int, int]: while True: private = 3 + r(PRIME - 4) if gcd(private, PRIME - 1) == 1: return private, pow(private, -1, PRIME - 1) def sign(subject: str, public: int, issuer: str, issuer_private: int, valid: bool) -> dict: s = {"subject": subject, "public": public, "issuer": issuer, "valid": valid} s["signature"] = pow(digest(f"{subject}|{public}|{issuer}|{valid}"), issuer_private, PRIME) return s def signature_valid(s: dict, issuer_public: int) -> bool: return pow(s["signature"], issuer_public, PRIME) == digest( f"{s['subject']}|{s['public']}|{s['issuer']}|{s['valid']}") def build_chain(length: int, name: str, r, anchor_private: int, valid: bool) -> list[dict]: top_private, top_name, chain = anchor_private, "anchor.test", [] for rung in range(length, 0, -1): private, public = key_pair(r) subject = name if rung == 1 else f"intermediate-{rung - 1}.test" chain.append(sign(subject, public, top_name, top_private, valid or rung > 1)) top_private, top_name = private, subject return chain def verify_chain(chain: list[dict], name: str, anchors: dict, counter: dict) -> bool: top_public = anchors.get(chain[0]["issuer"]) if top_public is None: return False for s in chain: counter["signature"] += 1 if not signature_valid(s, top_public) or not s["valid"]: return False top_public = s["public"] return chain[-1]["subject"] == name r = generator(SEED) anchor_private, anchor_public = key_pair(r) ANCHORS = {"anchor.test": anchor_public} submissions = [] for no in range(40): length, flaw = 1 + r(5), FLAWS[r(13)] name = "other.test" if flaw == "name" else "station.example" chain = build_chain(length, name, r, anchor_private, flaw != "validity") if flaw == "signature": chain[-1] = dict(chain[-1], signature=(chain[-1]["signature"] + 1) % PRIME) elif flaw == "anchor": chain[0] = dict(chain[0], issuer="unknown.test") submissions.append({"length": length, "flaw": flaw, "chain": chain}) def measure(submissions, per_round, verify=True): counter, correct, wrong, rounds = {"signature": 0}, 0, 0, 0 for s in submissions: given = (verify_chain(s["chain"], "station.example", ANCHORS, counter) if verify else True) correct += given == (s["flaw"] == "none") wrong += given != (s["flaw"] == "none") rounds += per_round * s["length"] return correct, wrong, rounds, counter["signature"] print(f"{'regime':<26s} {'correct':>7s} {'wrong':>7s} {'round trips':>12s} " f"{'signature checks':>17s}") for regime_name, per_round, do_verify in (("local anchor", 0, True), ("ask each rung remotely", 1, True), ("skip verification", 0, False)): c, w, t, i = measure(submissions, per_round, do_verify) print(f"{regime_name:<26s} {c:7d} {w:7d} {t:12d} {i:17d}") print() print(f"{'chain length':>13s} {'submissions':>12s} {'signature checks':>17s} " f"{'round trips':>12s}") for length in range(1, 6): matches = [s for s in submissions if s["length"] == length] print(f"{length:13d} {len(matches):12d} {measure(matches, 0)[3]:17d} {0:12d}") print() flaws = {k: sum(s["flaw"] == k for s in submissions) for k in ("none", "signature", "anchor", "name", "validity", "revoked")} print(f"submissions {len(submissions)}, decisions {len(submissions)}, " f"rungs {sum(s['length'] for s in submissions)}, flaws {flaws}") ``` ``` regime correct wrong round trips signature checks local anchor 33 7 0 98 ask each rung remotely 33 7 111 98 skip verification 17 23 0 0 chain length submissions signature checks round trips 1 9 9 0 2 10 18 0 3 10 24 0 4 3 12 0 5 8 35 0 submissions 40, decisions 40, rungs 111, flaws {'none': 17, 'signature': 3, 'anchor': 4, 'name': 5, 'validity': 4, 'revoked': 7} ``` The first two rows alone pay off the course's axis. The client walking with a local anchor got **33 decisions right**, 7 wrong, with **zero round trips.** The client asking each rung's issuer spent **111 round trips** and arrived at the **same** table: 33 correct, 7 wrong. Asking changed not a single decision; it only added 111 rounds. In the second table, the round-trip column is zero start to finish. Whether the chain is one rung or five, the client asks no one; the only thing that grows is **local work** — signature checks climb from 9 to 35. This is the difference between local work and rounds: one is paid on the client's processor, the other on the network, and the two cannot be converted into the same unit. Total signature checks are 98, rung count is 111. The gap is not an inconsistency but the walk itself: in the four chains that never reach the anchor, no signature is checked at all — the client sees at the first step that it has no anchor in hand — and when a broken rung is hit, the walk stops. Chain verification is an operation that **exits early**. #### What the Chain Does Not Say All 7 wrong decisions come from the same flaw, and its name is written in the table's last row: `revoked` 7. In these seven submissions the chain is consistent start to finish — signatures hold, the name is the one requested, the validity field is open — but the certificate is no longer counted valid. The distinction is this: chain verification measures the document's **internal consistency**, and everything it measures is written **inside** the document — which is why it can run locally and needs no round trip. Revocation, on the other hand, reports something happening **outside** the document and cannot be written into the document afterward, because the document freezes the moment it is signed. The chain verifies without asking; revocation cannot be known without asking. #### Configuration Flaws and Narrowings The table's third row is the cost of a flaw. A client that never runs chain verification gets away with zero round trips and zero signature checks; what it pays is correct decisions dropping from **33 to 17.** This is what is wrong — the work gained is local and cheap, what is lost is the decision itself. How this is done is not this course's subject. The parts of verification must also be counted separately, each with its narrowing written next to it: - **Name check.** In the measurement, 5 submissions were rejected for this check alone: their chains were flawless, their subjects did not satisfy the requested name. *Narrowing:* if the name check is built as an inseparable step of the verification path rather than a setting that can be turned on and off, it cannot be disabled. - **Validity check.** In the measurement, 4 submissions were rejected for this alone. *Narrowing:* if validity ranges are kept short, the window a wrong clock opens is also shortened. - **Width of the anchor set.** Every anchor in the client's hand makes **every** chain descending from that anchor acceptable. *Narrowing:* if the anchor set is narrowed to endpoints actually used, the set of acceptable chains narrows by the same proportion. - **The chain's intermediate rungs.** In the measurement, 4 submissions never reached the anchor and were rejected for that; the server sending incomplete intermediate certificates produces the same result. *Narrowing:* a missing rung is closed on the server's configuration, not on the client — not by widening the client's anchor set. The last item also explains a habit: most situations that look like a "certificate error" are not an attack, they are a **configuration flaw**. Where the flaw sits is separated by looking at which check fell. #### Summary - A certificate binds a name to a public key and puts the issuer's signature under that bond; the signature stands on the subject, key, issuer, and validity fields. - The signature does not make the document unchangeable, it makes a change show; it does not show the issuer was authorized, the subject still exists, or the key stayed in hand. - The chain stops at the trust anchor; what makes the anchor trustworthy is not its own signature but that it is in the client's hand. - Across forty submissions, the client walking with a local anchor took 33 correct and 7 wrong decisions with **0 round trips**; the client asking each rung spent **111 round trips** and arrived at the same table. - As chain length went from 1 to 5, local signature checks climbed from 9 to 35, round trips stayed at 0 on every row. - All 7 wrong decisions come from revocation: the chain measures the document's inside, revocation is what happens outside the document. #### Next Step Chain verification asked for no round at all, but it does not run in a vacuum: it happens inside a specific connection, at a specific moment, and the certificate reaches the client only after an exchange begins. That exchange has its own budget — how many messages go back and forth, at which round the certificate arrives, after which round the body starts being carried. The next lesson counts this budget and measures what happens to the intermediary's decision table once the budget closes. ### The TLS Handshake Source: https://academia.sh/en/courses/application-protocols/tls-handshake The handshake's round-trip budget, the round session resumption earns, and the 160 decisions the intermediary loses once the transport envelope closes. Chain verification asked for no round trip, but it was not done in a vacuum. Before the certificate reached the client, an exchange had to begin, the server had to present its chain, and the two sides had to arrive at a shared secret. **TLS (Transport Layer Security)** handshake does these three jobs inside a single setup. This lesson's question has two parts. The first is budget: how many round trips setup holds, who pays it, and how repeated payment is prevented. The second is the course's real subject: once setup completes, the message enters a **transport envelope** and the intermediary can no longer read what it says. What this does to the decision table counted through the course is measured here. #### What the Handshake Carries The handshake gathers everything that needs agreement into a small number of messages. ```text full setup client -> server : hello, supported suites, client's key share server -> client: hello, selected suite, server's key share, certificate chain, finished client -> server : finished, then application data session resumption client -> server : hello, previous session's resumption identity server -> client: hello, finished client -> server : finished, then application data ``` This listing is not run; it shows what the steps carry. Four jobs sit side by side: choosing the suite, exchanging key shares, presenting the certificate chain, and the two sides confirming setup is done. The four jobs are not independent: key shares cannot be interpreted before the suite is chosen, and finished cannot be computed before key shares are exchanged. This dependency sets the order of the steps; rounds can be merged but not swapped. The order is a design decision. The certificate chain arrives with the server's **first** response; the client makes no separate request to verify it. The previous lesson's measurement is the counterpart: chain verification ran with zero round trips, because everything it needed was already in hand. Resumption steps carry no certificate chain — the client has already verified that chain once and stored the result with the session. #### Choosing the Suite and Finishing The first of the four jobs is a negotiation: the client declares the **cipher suites** it supports, the server picks one. The suite decides, after setup, which symmetric transform and digest function will be used; the division of labor from the first lesson becomes concrete here. The negotiation has a problem repeated in every lesson of this course: the first two messages travel over a channel not yet protected, so a layer on the path can read the client's list. The specification's answer is the **finished** message at the last step of setup: it is computed over the digest of **all** handshake messages sent and received up to that point, using the key that comes out of setup. Both sides check the other's finished; if the handshake listing one side saw differs from the other's, finished does not hold and setup does not complete. This is encountered for the first time in the course. Every decision of the intermediary rested on a **claim**, and nothing verified it. Finished verifies the claim in the negotiation, making it checkable afterward. The difference is who checks it — not the intermediary, the **endpoints**. One thing finished does not check is the list's **content**: it verifies both sides saw the same list, not that no weak suite is on it. Keeping weak suites on the list is therefore a **configuration flaw**: the choice was made, the negotiation was verified, and setup still lands on a bad suite. *Narrowing:* if the list itself is narrowed — only acceptable suites are declared — the set the negotiation can land on also narrows and verification becomes meaningful. This is a symptom, and what is written here is what is wrong. #### The Budget Is Paid Per Connection Setup's round is written not to the exchange but to the **connection**: forty exchanges in a single connection pay setup once; in forty separate connections, forty times. The measurement below **models** full setup as two round trips and session resumption as one. These are model values chosen within the lesson; they differ across versions of the specification family, and the lesson makes no version claim. What is measured is not the numbers themselves but **how they grow with connection count.** ```python EXCHANGES = 40 FULL_ROUND = 2 # full setup budget modeled within the lesson RESUME_ROUND = 1 # session resumption budget modeled within the lesson print(f"{'connections':>12s} {'setup without resumption':>25s} {'setup with resumption':>22s} " f"{'rounds saved':>13s} {'total rounds':>13s} {'per exchange':>13s}") for connections in (40, 8, 4, 1): without = connections * FULL_ROUND with_resumption = FULL_ROUND + (connections - 1) * RESUME_ROUND total = with_resumption + EXCHANGES print(f"{connections:12d} {without:25d} {with_resumption:22d} {without - with_resumption:13d} " f"{total:13d} {total / EXCHANGES:13.3f}") ``` ``` connections setup without resumption setup with resumption rounds saved total rounds per exchange 40 80 41 39 81 2.025 8 16 9 7 49 1.225 4 8 5 3 45 1.125 1 2 2 0 42 1.050 ``` In the regime that opens forty separate connections, setup without resumption holds **80 rounds**; since the forty exchanges themselves hold 40 rounds, opening a connection is **twice** as expensive as doing the work. **Session resumption** gives back 39 rounds on this row, bringing the total to 81 — 2.025 rounds per exchange. Two observations read the table. First, the round resumption earns **depends on the connection count**: at one persistent connection the gain is zero, because there is nothing to resume. Second, resumption does not eliminate setup — it only **makes it cheaper**. Even on the forty-connection row, the per-exchange figure stays above 1.000; the only way to zero is keeping the connection persistent. Resumption earns something beyond rounds too, tying back to the previous lesson: in a resumed session the certificate chain is not re-presented, so chain verification is not redone either. A decision taken once is not retaken for as long as the session lives. Its cost sits in the same place: information about the chain **ages**, and the session does not see it. #### Once the Envelope Closes Once setup finishes, the application message enters the envelope. The intermediary sees the message arrive, sees its length, sees which host it is going to — but it **cannot see** the method, the path, the private marker, or the validator. The decision table counted through the course is now measured in two regimes. In the `open` regime the intermediary reads all five fields; in the `envelope` regime, only the host name. The oracle — the truth known because we produced the scenario — does not change. Forty exchanges and five decisions per exchange is a set of 200 decisions; the smallest difference measurable in this set is 1/200 = 0.0050. Assumptions of this lesson's two measurements: - **GT13** — In the budget table, full setup is **modeled within the lesson** as two round trips, session resumption as one; both values are chosen model values and make no specification version claim. - **GT14** — Setup's round is written to the connection, an exchange's round to the exchange: forty exchanges hold forty rounds, and in the resumed regime the first connection pays full setup, the rest pay resumption. - **GT15** — In the decision table the course's forty exchanges are reproduced from the same seed; five decisions per exchange are counted and the oracle does not change when the regime changes. - **GT16** — Two regimes are compared: in the `open` regime the intermediary reads the method, the path, the host name, the private marker, and the validator; in the `envelope` regime, only the host name. The host name is outside the envelope because it is declared in setup's first message, before a shared secret exists yet. - **GT17** — The intermediary sits on the path and looks only at what stays outside the envelope; every decision it cannot get is a **round debt** — one more round is thrown at the server on that exchange. - **GT18** — The bytes column is a count of the fields the intermediary can read, not the real message's length; the measurement is not a network measurement. ```python SEED = 20260809 SAFE = {"GET", "HEAD"} IDEMPOTENT = {"GET", "HEAD", "PUT", "DELETE"} METHODS = ["GET", "GET", "GET", "HEAD", "POST", "PUT", "DELETE"] PATHS = ["/measurement/north", "/measurement/slope", "/summary", "/record", "/session"] DECISIONS = ("cacheable", "shareable", "fresh", "repeatable", "routable") VISIBLE = {"open": ("method", "path", "host", "private_marker", "validator"), "envelope": ("host",)} def generator(seed: int): d = seed % 2147483646 + 1 def r(n: int) -> int: nonlocal d d = (d * 48271) % 2147483647 return d % n return r def exchanges(count: int = 40) -> list[dict]: r, items = generator(SEED), [] for i in range(count): method, path = METHODS[r(7)], PATHS[r(5)] private = path == "/session" or r(5) == 0 changed = r(3) == 0 items.append({"no": i + 1, "method": method, "path": path, "private": private, "changed": changed, "private_marker": private and r(4) != 0, "stamp_skipped": changed and r(6) == 0}) return items def oracle(a: dict) -> dict: return {"cacheable": a["method"] in SAFE and not a["private"], "shareable": not a["private"], "fresh": not a["changed"], "repeatable": a["method"] in IDEMPOTENT, "routable": True} def message(a: dict, visible: tuple) -> dict: full = {"method": a["method"], "path": a["path"], "host": "station.example", "private_marker": a["private_marker"], "validator": a["changed"] and not a["stamp_skipped"]} return {k: v for k, v in full.items() if k in visible} def intermediary(i: dict) -> dict: k = {} if "method" in i: k["repeatable"] = i["method"] in IDEMPOTENT if "private_marker" in i: k["shareable"] = not i["private_marker"] if "method" in i: k["cacheable"] = i["method"] in SAFE and not i["private_marker"] if "validator" in i: k["fresh"] = not i["validator"] if "host" in i: k["routable"] = True return k def bytes_seen(i: dict) -> int: return sum(len(k) + len(str(v)) + 4 for k, v in i.items()) def measure(name: str) -> tuple: correct = wrong = unavailable = extra_round = total_bytes = 0 for a in exchanges(): truth, given = oracle(a), intermediary(message(a, VISIBLE[name])) missing = False for decision in DECISIONS: if decision not in given: unavailable += 1 missing = True elif given[decision] == truth[decision]: correct += 1 else: wrong += 1 extra_round += 1 if missing else 0 total_bytes += bytes_seen(message(a, VISIBLE[name])) return correct, wrong, unavailable, extra_round, total_bytes print(f"{'regime':<8s} {'correct':>7s} {'wrong':>7s} {'unavailable':>12s} " f"{'extra round':>13s} {'bytes':>6s}") for name in ("open", "envelope"): print(("{:<8s} {:7d} {:7d} {:12d} {:13d} {:6d}").format(name, *measure(name))) print() print(f"{'decision':<20s} {'open':>8s} {'envelope':>10s}") for decision in DECISIONS: counts = [sum(decision in intermediary(message(a, VISIBLE[name])) for a in exchanges()) for name in ("open", "envelope")] print(f"{decision:<20s} {counts[0]:8d} {counts[1]:10d}") ``` ``` regime correct wrong unavailable extra round bytes open 195 5 0 0 3847 envelope 40 0 160 40 920 decision open envelope cacheable 40 0 shareable 40 0 fresh 40 0 repeatable 40 0 routable 40 40 ``` #### Where the Hundred Sixty Decisions Went In the `open` regime the intermediary took 200 of 200 decisions and never returned to the server; 195 were correct, 5 wrong. In the `envelope` regime **160 decisions become unavailable.** These decisions did not disappear; because the intermediary could not get them, they **moved to the endpoints.** The decision-maker was not just the two endpoints before; now it is. The second table shows exactly where this 160 was gathered from. Four of the five decisions — `cacheable`, `shareable`, `fresh`, `repeatable` — are taken in all forty of the forty exchanges in the open regime and in none of them in the envelope regime: four decisions times forty exchanges, 160. The one decision still standing is `routable`, because its only support, the host name, stays outside the envelope. The loss is not spread evenly, it is **sharp**: hiding one field does not reduce the decision tied to that field, it zeroes it. The extra-round column shows the price of this: in **all forty** of the 40 exchanges the intermediary is missing at least one decision, meaning all forty throw one more round at the server. The visibility the intermediary loses turns directly into a round debt. The bytes column looks reversed at first glance: it drops from 3847 to 920. The message did not shrink; only the fields the intermediary can **read** shrank. This is the reverse reading of the course's second claim — there, adding a field did not earn a decision; here, hiding a field earns bytes but costs decisions. The real line is the wrong column: **0 in the envelope regime.** None of the remaining 40 decisions are wrong. The reason is direct: the only decision left is `routable`, and that decision looks only at the host name; since the host name stays outside the envelope, it is always correct. The 5 wrong decisions in the open regime came from the places where the intermediary **believed a claim** — a marker the server forgot to set, and a change the validator missed. The intermediary cannot believe a field it cannot read. The sentence that follows is the course's third claim, not a description of a flaw: **fewer decisions, but none wrong.** The envelope regime lets the intermediary take 40 of 200 decisions; in return it drops the intermediary's error surface to zero. Which is preferable is not a measurement question but a design decision. #### The Remaining Surface and Its Narrowings The envelope is a record format: the application message is split into pieces, each wrapped with its length and carried that way. This draws the boundary of what the intermediary sees: it sees the record boundaries and lengths — because it cannot forward the message without seeing them — but not what is inside the record. The zeroing of the 160 in the decision table is the result of this distinction: what is left in the intermediary's hand is the message's **container**, not its content. The envelope does not close everything. What the intermediary still sees, and the narrowing for each: - **Host name.** Declared in setup's first message, before a shared secret exists yet; in the measurement it is `routable`'s only support. *Narrowing:* mechanisms exist where the name is also folded into setup's protected part; in that case the table's last row also moves to unavailable and the intermediary loses the routing decision too. - **Certificate chain.** In full setup the chain is presented in the open and carries the subject's name. *Narrowing:* in resumed sessions the chain is never presented; as the connection count drops, this surface shrinks too. - **Message length and timing.** The envelope closes the body, it does not close its length or arrival moment. *Narrowing:* padding and merging push the length into a band. There is exactly one way for an intermediary to regain the 160 decisions, and it is not a loophole but a placement decision: the intermediary **terminates the envelope itself.** Then there are two separate setups — one between the client and the intermediary, one between the intermediary and the server — and the intermediary in the middle returns to the open regime, regaining the 200 decisions in the table's first row. The price is not read from the table but from the definition: the endpoints no longer set up with each other, both now set up with the intermediary, and the chain the client verifies is the intermediary's chain, not the server's. *Narrowing:* such an intermediary's anchor must be separately placed on the client; until it is, chain verification falls and the decision loss stays where it was. There is also a common **configuration flaw**: completing setup and passing through without verifying the certificate chain. The gain is local and was counted in the previous lesson's measurement — not a single round. What is paid is not knowing **whom** the bond the envelope protects was made with. What this lesson states is what is wrong; how this is done is not written. #### Summary - The handshake gathers suite selection, key-share exchange, certificate-chain presentation, and mutual finishing into a single setup; because the chain arrives with the first response, its verification needs no separate round. - The suite negotiation happens on a still-unprotected channel; because the finished message stands on the digest of the whole handshake, the negotiation is checked by the endpoints, not the intermediary. - Setup's budget is written to the connection, not the exchange: 80 rounds without resumption at 40 connections, 41 with resumption; 2 rounds at one persistent connection. - Session resumption earns 39 rounds in the forty-connection regime and does not repeat chain verification; its gain is zero at one connection. - Once the envelope closes, 160 of 200 decisions move from the intermediary to the endpoints, all forty of the forty exchanges incur a round debt, and the bytes the intermediary reads drop from 3847 to 920. - **None of the remaining 40 decisions are wrong**; the 5 wrong decisions in the open regime came from where the intermediary believed a claim, and the intermediary cannot believe a field it cannot read. #### Next Step Chain verification ran locally, the handshake paid its budget per connection, and once the envelope closed 160 decisions moved to the endpoints. One thing alone stayed unmeasured: the chain the client verified was valid **at the moment** it was verified. A certificate has a start and an end, it can be revoked between them, and revocation cannot be written into the document. The next lesson counts the certificate's life cycle and what revocation adds to the round budget: the chain verified without asking; revocation cannot be known without asking. ### Certificate Life Cycle Source: https://academia.sh/en/courses/application-protocols/certificate-life-cycle Revocation as a round debt: how many exchanges not asking costs, the round asking adds per exchange, the knowledge delay of the options in between, and the symptoms of misconfiguration seen at the client. The previous lesson closed the transport envelope and counted what the intermediary lost. Underneath the closed envelope stood one assumption: the client had verified the server's chain and built the setup on it. That verification was valid at the **moment** it was made. A certificate has a start and an end; between the two it can be revoked. Revocation cannot be written into the document, because the document freezes the moment it is signed — anything written afterward breaks the signature. This lesson's question follows from that: how is something unwritable in the document known, and what does knowing it cost? Chain verification ran **without asking**; revocation cannot be known **without asking**. The difference between the two is a round debt. #### The Certificate's Life and Where Revocation Sits Life is the span between the two timestamps in the certificate's validity field, sitting **inside** the document, under the signature. That is why the validity check is local: the client looks at its own clock and asks no one. **Renewal** is issuing a new document before life runs out. It does not extend the old document — extending it would mean changing a signed field. A new document is produced, the old document stays valid until its own end, and the two live side by side for a while. If the key has changed, the old document says nothing about the new key. **Revocation** is declaring the binding invalid before its life runs out. Its causes sit outside the document: the private key falling into other hands, the name changing ownership, the binding having been set up wrong from the start. None of these can be written into the document afterward, so the issuer publishes a separate record. ```text life cycle request -> signing -> distribution -> use -> renewal -> ... -> expiration | +-- revocation: at any moment, from outside the document fields that carry life serial number : names the document uniquely within the issuer validity : start .. end <- life is this span issuer : intermediate-1.test <- the party that will publish the revocation revocation record list issuer : intermediate-1.test issue moment : the moment this list was produced next issue : the moment the list will be refreshed <- upper bound on knowledge delay entries : serial number | revocation moment | reason ``` This listing is not executed; it shows what the fields declare. The record names the document by serial number without changing it — it drops a note **outside** it. The asymmetry sits here: validity is read from inside the document, revocation from outside it. The first needs no round trip, the second does. #### The Setup of the Measurement The setup matches the rest of the course: the North Slope measurement station presents a measurement to a client, and on every exchange the client decides whether the presented certificate is still valid. - **GT1.** The measurement window is 960 hours and forty exchanges happen inside it; exchange moments are drawn from the generator within the window. - **GT2.** The revocation moment is tried separately at every hour of the window: 960 moments are swept, and forty exchanges are evaluated for each moment. - **GT3.** The oracle — the truth behind the decision — is known because we produced the scenario: the certificate is invalid on every exchange after the revocation moment. - **GT4.** The time until the client learns of the revocation is the **knowledge delay**; every exchange that falls within that span is a **false acceptance**. - **GT5.** The revocation list's refresh interval is 24 hours, the presented response's freshness is 6, the short-lived certificate's life is 12, and the long-lived one's is 2160; all four are model values chosen within the lesson, with no version claim. - **GT6.** The chain is three rungs — the leaf and two intermediate certificates; the trust anchor is not asked. - **GT7.** The round is only the round trip the **client** pays; a round trip paid by the server or the record's publisher does not enter this column. - **GT8.** No connection is opened, no certificate is resolved; the listings are the scenario. At a single revocation moment the set is forty decisions, resolution 1/40 = 0.025; the average columns are drawn from all 960 moments — 38,400 decisions total. #### The Cost of Not Asking and Asking ```python SEED = 20260809 WINDOW = 960 # hours: forty-day measurement window LIFE = 2160 # hours: the long-lived certificate's modeled life RUNGS = 3 # leaf plus two intermediate certificates; the anchor is not asked def generator(seed: int): d = seed % 2147483646 + 1 def r(n: int) -> int: nonlocal d d = (d * 48271) % 2147483647 return d % n return r generate = generator(SEED) TIMES = sorted(generate(959) for _ in range(40)) REGIMES = ( # name, round per exchange, refresh interval (hours), client fetches it itself ("not asking", 0, None, False), ("asking every exchange", 1, 0, True), ("pre-distributed revocation list", 0, 24, True), ("response presented with certificate", 0, 6, False), ("short-lived certificate", 0, 12, False), ) def knowledge_moment(revoked_at: int, interval) -> int: if interval is None: return LIFE if interval == 0: return revoked_at return revoked_at + interval - revoked_at % interval def measure(interval) -> tuple[float, int, float, int]: delay_total = false_total = delay_worst = false_worst = 0 for revoked_at in range(WINDOW): known_at = knowledge_moment(revoked_at, interval) false_count = sum(1 for t in TIMES if revoked_at <= t < known_at) delay_total, false_total = delay_total + known_at - revoked_at, false_total + false_count delay_worst = max(delay_worst, known_at - revoked_at) false_worst = max(false_worst, false_count) return delay_total / WINDOW, delay_worst, false_total / WINDOW, false_worst def client_rounds(per_exchange: int, interval, fetches: bool, count: int, rungs: int) -> int: rounds = per_exchange * count * rungs return rounds + (WINDOW // interval * rungs if fetches and interval else 0) print(f"{'regime':<38s}{'round/exchange':>16s}{'client rounds':>15s}" f"{'knowledge delay':>18s}{'false acceptance':>19s}") for name, per_exchange, interval, fetches in REGIMES: dg, dw, fg, fw = measure(interval) print(f"{name:<38s}{per_exchange:16d}" f"{client_rounds(per_exchange, interval, fetches, len(TIMES), RUNGS):15d}" f"{f'{dg:.1f} / {dw}':>18s}{f'{fg:.2f} / {fw}':>19s}") print() print(f"{'rungs':>7s}{'ask every exchange':>21s}{'revocation list':>18s}" f"{'presented response':>21s}") for h in range(1, 6): row = [client_rounds(b, a, c, len(TIMES), h) for _, b, a, c in REGIMES[1:4]] print(f"{h:7d}{row[0]:21d}{row[1]:18d}{row[2]:21d}") print() print(f"{'exchanges':>10s}{'ask every exchange':>21s}{'revocation list':>18s}") for n in (40, 120, 400, 4000): row = [client_rounds(b, a, c, n, RUNGS) for _, b, a, c in REGIMES[1:3]] print(f"{n:10d}{row[0]:21d}{row[1]:18d}") print() print(f"exchanges {len(TIMES)}, revocation moments {WINDOW}, " f"decisions {len(TIMES) * WINDOW}") ``` ``` regime round/exchange client rounds knowledge delay false acceptance not asking 0 0 1680.5 / 2160 22.02 / 40 asking every exchange 1 120 0.0 / 0 0.00 / 0 pre-distributed revocation list 0 120 12.5 / 24 0.57 / 4 response presented with certificate 0 0 3.5 / 6 0.15 / 2 short-lived certificate 0 0 6.5 / 12 0.31 / 3 rungs ask every exchange revocation list presented response 1 40 40 0 2 80 80 0 3 120 120 0 4 160 160 0 5 200 200 0 exchanges ask every exchange revocation list 40 120 120 120 360 120 400 1200 120 4000 12000 120 exchanges 40, revocation moments 960, decisions 38400 ``` The first table's first two rows are the lesson's two extremes. A client that never asks spends **zero rounds** and treats a revoked certificate as valid for an average of **22.02 exchanges**; in the worst case, for all forty. The reason is not subtle: not asking leaves only the document's own end as a source of information, and that end sits far past the window. A client that asks on every exchange brings false acceptance down to **zero**. The 120 rounds it pays come from two factors: forty exchanges times three rungs. The round per exchange looks like 1, but **as many questions as there are rungs** are asked, because each rung of the chain can be revoked separately. The second table unpacks that factor and sharpens the distinction at the course's axis. In the Certificates and the Trust Chain lesson, chain verification's round was **independent** of chain length; the revocation question's round is **directly proportional** to it: 40 at one rung, 200 at five. The document's inside arrives with the chain; the outside does not, and is asked for separately for every rung. The third table shows the divide. Asking's round grows linearly with the exchange count — 120 at forty exchanges, 12,000 at four thousand — but the pre-distributed list's round **stays fixed**: 120. The list is paid once per window, the question once per exchange. That the two come out equal on the forty-exchange row is a coincidence; the distinction is not in the total but in **where the round is written.** #### The Options in Between The table's last three rows fill the space between the two extremes, sharing one property: **the round per exchange is zero.** None of them adds a round trip to the exchange's path; where they diverge is the knowledge delay. The **pre-distributed revocation list** refreshes once a day; the delay averages 12.5 hours, 24 in the worst case, and an average of 0.57 exchanges is falsely accepted. The **response presented with the certificate** is a validity response from the issuer sent along with the certificate during setup; the client's round is zero, because the response is already inside a message that was going to arrive anyway. The delay drops to 3.5 hours, false acceptance to 0.15. The **short-lived certificate**, in turn, discards the revocation mechanism entirely: the document lives so briefly it never needs revoking, and the delay is the remaining life — an average of 6.5 hours, false acceptance 0.31. Reading the three rows side by side, a pattern emerges. At a delay of 3.5, false acceptance is 0.15; at 6.5, it is 0.31; at 12.5, it is 0.57. False acceptance grows with the delay, roughly in proportion to it. So even though the mechanisms carry different names, what is measured is a single number: **choosing a mechanism is choosing a delay.** The revocation question itself is a surface that has to be counted too: it tells the answering party which document is being looked at, and asking on every exchange also reveals the exchange frequency. *Narrowing:* when the response is presented together with the certificate, the client never asks, so there is no notification either; the pre-distributed list carries the whole list rather than individual documents, so it reveals nothing about which document is being looked at. Not asking's surface sits in the table's first row — an average of 22.02 exchanges. *Narrowing:* the only thing that narrows that window is the document's life. #### As Life Shortens ```python SEED = 20260809 WINDOW = 960 # hours: forty-day measurement window LIFETIMES = (2160, 720, 168, 72, 24, 12, 6) # hours def generator(seed: int): d = seed % 2147483646 + 1 def r(n: int) -> int: nonlocal d d = (d * 48271) % 2147483647 return d % n return r generate = generator(SEED) TIMES = sorted(generate(959) for _ in range(40)) def measure(life: int) -> tuple[float, float, int]: delay_total = false_total = false_worst = 0 for revoked_at in range(WINDOW): known_at = revoked_at + life - revoked_at % life false_count = sum(1 for t in TIMES if revoked_at <= t < known_at) delay_total, false_total = delay_total + known_at - revoked_at, false_total + false_count false_worst = max(false_worst, false_count) return delay_total / WINDOW, false_total / WINDOW, false_worst print(f"{'life (hours)':>13s}{'renewals':>10s}{'avg delay':>13s}" f"{'false acceptance':>19s}{'renewals x life':>17s}") for life in LIFETIMES: dg, fg, fw = measure(life) renewals = WINDOW // life print(f"{life:13d}{renewals:10d}{dg:13.1f}{f'{fg:.2f} / {fw}':>19s}" f"{renewals * life:17d}") ``` ``` life (hours) renewals avg delay false acceptance renewals x life 2160 0 1680.5 22.02 / 40 0 720 1 420.5 13.02 / 28 720 168 5 87.5 3.82 / 8 840 72 13 37.1 1.32 / 4 936 24 40 12.5 0.57 / 4 960 12 80 6.5 0.31 / 3 960 6 160 3.5 0.15 / 2 960 ``` This table has no revocation mechanism; the client learns that something has become invalid only when the document ends. The two columns move in opposite directions. At a life of 2160 hours, no renewal happens within the window, the delay averages 1680.5 hours, and 22.02 exchanges are falsely accepted. When life drops to 6 hours, the delay falls to 3.5 hours and false acceptance to 0.15 — but 160 renewals have to happen within the window. The last column shows the total work is conserved: on every row where life is shorter than the window, renewal count times life equals the window. The delay gained is inversely proportional to the renewal count paid. 160 renewals in forty days means one roughly every six hours; that is not work done by hand. **A short life is a condition for automation**, and if the condition is not met, the gain turns into a cost. One more observation is not a coincidence. This table's 24-, 12-, and 6-hour rows give the **same** numbers as the previous table's revocation-list, short-lived-certificate, and presented-response rows. Three separate mechanisms are a function of a single parameter. The mechanism's name does not change the false acceptance count; the only thing it changes is **who pays the round, when, and how many times.** #### Symptoms of Misconfiguration Verification is not a single decision but a walk, and each step of the walk can fail separately. The symptom gives away which step failed. ```text verification walk and the steps it can fail at 1 the presented chain was collected <- a missing intermediate certificate fails here 2 every rung's signature was verified 3 did the chain reach the trust anchor <- a self-signed chain fails here 4 is the validity interval open <- an expired certificate fails here 5 does the leaf's name satisfy the requested name <- a name mismatch fails here 6 has one of the rungs been revoked <- only this step asks outside the document ``` - **Expired certificate.** Symptom: verification fails at the fourth step and the failure begins **at a single moment**, because the interval is the same for everyone. The distinguishing mark: the failure does not vary with network, path, or client. *Narrowing:* if renewal is triggered before the end, the interval never closes; as life shortens, this trigger has to become automated. - **Missing intermediate certificate.** Symptom: the same server works on some clients and not on others. A client that acquired and cached the missing rung elsewhere can complete the chain; one that did not cannot. The distinguishing mark is that the outcome **varies by client.** *Narrowing:* the missing rung is closed by adding it to the chain the server presents, not by widening the client's anchor set. - **Name mismatch.** Symptom: the signatures hold, the chain reaches the anchor, the validity interval is open, and verification still fails at the fifth step. The distinguishing mark: the failure depends on **which name** was used. *Narrowing:* if the name set the certificate declares and the name set the server answers to come from a single source, the two cannot diverge. - **Self-signed chain.** Symptom: the chain is a single rung and that rung's issuer is itself; without a match in the client's anchor set, it fails at the third step. The distinguishing mark is that the failure is **not seen** on a client that has placed the anchor by hand. This does not mean a self-signed document is flawed — trust anchors themselves are mostly self-signed too; the distinction is whether the document is in the client's hand. *Narrowing:* if an anchor is placed, its scope is narrowed to a single endpoint, because every anchor makes every chain descending from it acceptable. All four are the failure of **one step** of verification. Turning verification off entirely silences all four symptoms at once, and because it silences them, it is not a fix — the Certificates and the Trust Chain lesson measured that a client that never verifies has a lower count of correct decisions. This lesson writes down what is wrong, not how to do it. #### Summary - Life sits inside the document and is checked locally; revocation sits outside the document and, because it cannot be written into a signed field, is published as a separate record. - Renewal does not extend the old document, it issues a new one; the two live side by side for a while, and if the key has changed, the old document says nothing about the new key. - A client that never asks spends zero rounds and treats a revoked certificate as valid for an average of 22.02 exchanges; a client that asks on every exchange zeroes out false acceptance and pays 120 rounds. - The revocation question's round is directly proportional to chain length — 40 at one rung, 200 at five — and grows linearly with exchange count, while the pre-distributed list's round stays fixed. - The three options in between have zero round per exchange, and their false acceptance depends only on the knowledge delay: 0.15 at 3.5 hours, 0.31 at 6.5, 0.57 at 12.5. - As life shortens, the delay drops but renewals rise; renewal count times life is fixed to the window, so the delay gained is inversely proportional to the renewals paid. #### Next Step Secure Transport closes here. Key roles were split, the chain was verified locally, setup's round was counted, and revocation's round debt was paid. The loss measured in the TLS Handshake lesson still stands too: once the envelope closed, 160 decisions had moved from the intermediary to the endpoints. But in the middle of that loss, the connection was still shaped as **request–response**: the client sends a message, the server returns one, and the turn passes back to the client. Even though the intermediary could not see inside the message, it knew how many messages there were and where each one started and ended. The next topic asks: if the connection's **shape** changes too, what remains? ## Other Application Protocols Two-way persistent connections, one-way server streaming, the cost of polling, the channel split in file transfer, the submission/retrieval split in mail, clock synchronization and device management. ### WebSocket Source: https://academia.sh/en/courses/application-protocols/websocket The upgrade handshake as a regime change: the intermediary falls into a tunnel after 101, all two hundred decisions move to the endpoints, none of the fields the frame carries yields a decision, and masking has a byte cost. The previous lesson closed the transport envelope and counted the intermediary's loss: one hundred sixty of the two hundred decisions moved out to the endpoints, leaving only the forty decisions derived from the host name. Despite the size of the loss, one thing had stayed in place. The connection was still shaped as **request–response**: the client sends a message, the server returns one, and the turn passes back to the client. Even though the intermediary could not see inside the message, it knew how many messages there were and where each one started and ended. This lesson's question is: if the connection's **shape** changes too, what remains? **WebSocket** does exactly this. It starts as an ordinary HTTP request, switches to its own frame format with a single response, and from that point on the connection carries no request line, no header field, no status code. Either end can write to the other whenever it wants, over the same connection. #### The Upgrade Handshake The connection opens with a single exchange the intermediary can read. The client sends an ordinary GET request; what sets the request apart is the header fields declaring that the connection is about to be **upgraded**. This is called the **upgrade handshake**. ```text GET /stream HTTP/1.1 Host: station.example Upgrade: websocket Connection: Upgrade Sec-WebSocket-Key: Sec-WebSocket-Version: 13 HTTP/1.1 101 Switching Protocols Upgrade: websocket Connection: Upgrade Sec-WebSocket-Accept: ``` The `Sec-WebSocket-Accept` value is derived from the client's key together with a string fixed in the specification. This derivation has **no confidentiality function**; it only ties the response to an end that actually understood the upgrade: an end that does not know the fixed string in the specification — a script sending an ordinary request, say — cannot produce the right accept value. The specification's defence against poisoning an intermediary's cache is not this field but the masking of every frame sent from client to server. The accept value's computation and the server-side state it sets up were written in the Caching, Queues and Asynchronous Processing course, where what was measured was the state the server holds per connection. **What is measured here is the decision the intermediary in between can take from the message.** The decisive line is `101 Switching Protocols`. The 100 class is an interim response: the request is still in progress, no result yet. 101, instead, does not report the outcome of a process — it announces that **a different protocol will now be spoken.** After this line there is no HTTP on the connection. #### The Intermediary's Measure The course's setup is the same in this lesson too. The North Slope measurement station serves a reading to a client; between the two stands an intermediary that does not know the application and decides only from what is written in the message. - **OP1.** Forty exchanges and five decisions per exchange are measured: `storable`, `shareable`, `fresh`, `repeatable`, `redirectable`. The set is two hundred decisions. - **OP2.** The oracle — the truth behind the decision — is known because we produced the setup ourselves. The intermediary's answer is compared against this truth; every exchange it has to ask about is a **round debt**. - **OP3.** In the `open` regime the intermediary reads the method, path, host name, the private flag and the validator. In the `tunnel` regime it reads none of them. - **OP4.** All forty exchanges are carried over a single WebSocket connection; the connection opens once and does not close. - **OP5.** The dumps and frame sizes are the setup; no connection is opened anywhere in the lesson. ```python SEED = 20260809 SAFE = {"GET", "HEAD"} IDEMPOTENT = {"GET", "HEAD", "PUT", "DELETE"} METHOD_POOL = ["GET", "GET", "GET", "HEAD", "POST", "PUT", "DELETE"] PATH_POOL = ["/measurement/north", "/measurement/slope", "/summary", "/register", "/session"] DECISIONS = ("storable", "shareable", "fresh", "repeatable", "redirectable") VISIBLE = {"open": ("method", "path", "host", "private_marker", "validator"), "tunnel": ()} def generator(seed: int): d = seed % 2147483646 + 1 def r(n: int) -> int: nonlocal d d = (d * 48271) % 2147483647 return d % n return r def exchanges(count: int = 40) -> list[dict]: r, out = generator(SEED), [] for i in range(count): method, path = METHOD_POOL[r(7)], PATH_POOL[r(5)] private = path == "/session" or r(5) == 0 stale = r(3) == 0 out.append({"no": i + 1, "method": method, "path": path, "private": private, "stale": stale, "private_marker": private and r(4) != 0, "tag_missed": stale and r(6) == 0}) return out def oracle(a: dict) -> dict: return {"storable": a["method"] in SAFE and not a["private"], "shareable": not a["private"], "fresh": not a["stale"], "repeatable": a["method"] in IDEMPOTENT, "redirectable": True} def message(a: dict, visible: tuple) -> dict: full = {"method": a["method"], "path": a["path"], "host": "station.example", "private_marker": a["private_marker"], "validator": a["stale"] and not a["tag_missed"]} return {k: v for k, v in full.items() if k in visible} def infer(i: dict) -> dict: k = {} if "method" in i: k["repeatable"] = i["method"] in IDEMPOTENT if "private_marker" in i: k["shareable"] = not i["private_marker"] if "method" in i: k["storable"] = i["method"] in SAFE and not i["private_marker"] if "validator" in i: k["fresh"] = not i["validator"] if "host" in i: k["redirectable"] = True return k def measure(name: str) -> tuple: correct = wrong = unavailable = extra_round = total_bytes = 0 for a in exchanges(): truth, given = oracle(a), infer(message(a, VISIBLE[name])) missing = False for decision in DECISIONS: if decision not in given: unavailable, missing = unavailable + 1, True elif given[decision] == truth[decision]: correct += 1 else: wrong += 1 extra_round += 1 if missing else 0 total_bytes += sum(len(k) + len(str(v)) + 4 for k, v in message(a, VISIBLE[name]).items()) return correct, wrong, unavailable, extra_round, total_bytes print(f"{'regime':<7s} {'correct':>7s} {'wrong':>6s} {'unavailable':>11s} " f"{'round debt':>10s} {'bytes':>6s}") for name in ("open", "tunnel"): c, w, u, e, b = measure(name) print(f"{name:<7s} {c:7d} {w:6d} {u:11d} {e:10d} {b:6d}") ``` ``` regime correct wrong unavailable round debt bytes open 195 5 0 0 3861 tunnel 0 0 200 40 0 ``` Before the upgrade, the intermediary read all forty of the forty exchanges and took two hundred of the two hundred decisions on its own, with no extra round trip to the server. After the upgrade, the number of decisions left in its hands is **zero**, and all forty of the forty exchanges turn into a round debt. The gap between the two rows is larger than the transport envelope's gap. In the envelope regime the intermediary still saw the host name and could take the `redirectable` decision on all forty; the tunnel loses that one too. The reason is not encryption alone: **after 101, no field that could feed the `redirectable` decision is ever visible on the connection again.** The routing decision was taken once, at the moment the connection was set up, and it is frozen after that. #### The Handshake Is the Last Message the Intermediary Reads The upgrade handshake is not an exception — it is a **boundary.** Running the messages carried over the connection through the same decision function, one by one, shows this. ```python IDEMPOTENT = {"GET", "HEAD", "PUT", "DELETE"} SAFE = {"GET", "HEAD"} def infer(i: dict) -> dict: # same decision function as the previous block k = {} if "method" in i: k["repeatable"] = i["method"] in IDEMPOTENT if "private_marker" in i: k["shareable"] = not i["private_marker"] if "method" in i: k["storable"] = i["method"] in SAFE and not i["private_marker"] if "validator" in i: k["fresh"] = not i["validator"] if "host" in i: k["redirectable"] = True return k CONNECTION = [ ("upgrade request", {"method": "GET", "path": "/stream", "host": "station.example", "upgrade": "websocket"}), ("101 response", {"upgrade": "websocket", "accept": "test-value"}), ("text frame", {"fin": 1, "opcode": 1, "masked": True, "length": 96}), ("close frame", {"fin": 1, "opcode": 8, "masked": True, "length": 2}), ] print(f"{'message':<18s} {'fields':>6s} {'decisions':>9s} decisions taken") for name, m in CONNECTION: k = infer(m) print(f"{name:<18s} {len(m):6d} {len(k):9d} {', '.join(k) or '-'}") ``` ``` message fields decisions decisions taken upgrade request 4 2 repeatable, redirectable 101 response 2 0 - text frame 4 0 - close frame 4 0 - ``` The upgrade request gives **two** of the five decisions: `repeatable` from the method field, `redirectable` from the host field. The remaining three need fields from the response, and that response never comes — what comes back is 101, and it carries neither the private flag nor the validator. These two decisions are the only ones the intermediary holds for the entire life of the connection. #### The Frame Is Visible, Yet No Decision Follows In the table above, the frames carry **four fields** and still produce zero decisions. This says the loss does not come from encryption alone. The frame's layout is fixed in the specification, and every field it carries is **about the transport**, not about the message. ```text 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-------+-+-------------+-------------------------------+ |F|R|R|R| opcode|M| length | extended payload length | |I|S|S|S| (4bit)|A| (7 bit) | if the length field is 126 | |N|V|V|V| |S| | | | |1|2|3| |K| | | +-+-+-+-+-------+-+-------------+-------------------------------+ | masking key (32 bit, if the mask bit is 1) | +---------------------------------------------------------------+ | payload data | +---------------------------------------------------------------+ ``` **Framing** is the rule that splits a byte stream into message boundaries. The length field is seven bits; payloads up to 125 are written directly, a value of 126 declares that a two-byte length field follows, and 127 declares an eight-byte one. When the mask bit is set, a four-byte key is added and the payload is folded with it. The mask is mandatory in the client direction. ```python OPCODES = {0: "continuation", 1: "text", 2: "binary", 8: "close", 9: "ping", 10: "pong"} def header_size(payload: int, masked: bool) -> int: n = 2 # FIN, RSV, opcode, mask bit, length n += 8 if payload > 65535 else 2 if payload > 125 else 0 # extended length n += 4 if masked else 0 # masking key only in the client direction return n print("opcodes:", ", ".join(f"{k}={v}" for k, v in OPCODES.items())) print() print(f"{'payload':>7s} {'direction':<18s} {'header':>7s} {'total':>7s} {'header share':>13s}") for payload, masked in ((2, True), (96, True), (125, True), (126, True), (96, False), (70000, False)): h = header_size(payload, masked) direction = "client to server" if masked else "server to client" print(f"{payload:7d} {direction:<18s} {h:7d} {h + payload:7d} {h / (h + payload):13.3%}") ``` ``` opcodes: 0=continuation, 1=text, 2=binary, 8=close, 9=ping, 10=pong payload direction header total header share 2 client to server 6 8 75.000% 96 client to server 6 102 5.882% 125 client to server 6 131 4.580% 126 client to server 8 134 5.970% 96 server to client 2 98 2.041% 70000 server to client 10 70010 0.014% ``` In the open regime, the 3861 bytes the intermediary reads spread over forty exchanges come to 96 bytes per exchange; the frame header wraps the same payload, in the server direction, in **two bytes.** The gain is real and shows up clearly on small messages. But its counterpart is measured just as precisely: 96 bytes buy two hundred decisions, two bytes buy zero. This is the course's second claim read in reverse. There, adding a field bought no decision: three more fields, more bytes, zero new decisions. Here the same thing shows up the other way around — **cutting bytes does not cost a decision either; what costs a decision is the field itself disappearing.** There is no link between bytes and decisions; the link is between fields and decisions. #### The One Lever Left in the Intermediary's Hand The intermediary cannot take decisions, but it keeps carrying the connection, and it retains exactly one authority over what it carries: **closing it.** Cutting an idle connection after a fixed span — the **idle timeout** — is the ordinary rule an intermediary reaches for to bound the number of connections it keeps open. In the open regime this rule is harmless; in request–response shape the connection is already idle between messages and can be reopened. In the tunnel, the connection itself is the carrier of state: cutting it takes with it whatever the two ends had agreed on. The specification answers this with two **control frames**. The **ping** and **pong** opcodes may carry application data — a pong returns the exact bytes the ping carried — but for this lesson their job is to move bytes over the connection and confirm that both ends are still up. They **tell** the intermediary nothing — as measured in the table, they produce no decision — they only reset the intermediary's idle counter. That the intermediary's decision count from the message stays zero while a design constraint on the connection still follows from it is this lesson's quiet result. One more field can be carried in the upgrade request: the `Sec-WebSocket-Protocol` header names the rule by which the frames will be interpreted, and the server repeats the name it chose in the response. This name is the only **meaning-related mark** the intermediary will see on the connection, and it too passes only once, in the opening exchange. After that, the intermediary carries a stream whose name it knows but whose content it cannot read. #### Where the Decisions Move Two hundred decisions do not vanish, they move. In the tunnel regime each one now has to be taken at an endpoint, and that means work written into the application. | Decision | Whose in the open regime | Whose in the tunnel | What it requires at the endpoint | |---|---|---|---| | `repeatable` | the intermediary | the sending end | a message identifier and a retry rule | | `storable` | the intermediary | the receiving end | an in-application copy and invalidation | | `shareable` | the intermediary | the server | writing the scope into the frame | | `fresh` | the intermediary | the receiving end | a version or counter field | | `redirectable` | the intermediary | the connection setup | once at opening, then fixed | The table's right column is what an application using WebSocket has to **rewrite**. In HTTP, all of these decisions were defined inside the specification, and any intermediary along the path could apply them. In the frame format, the only thing defined is the transport; the fields that carry meaning are agreed between the two ends themselves. It is also worth recording that this is not a defect. In the measurement table, the tunnel row's `wrong` column is **zero.** The intermediary takes no wrong decision, because it takes no decision at all. The open regime's five wrong decisions were the price of trusting a declaration; the tunnel does not pay that price either. The choice is between taking the decision cheaply on the path, or taking it fully but expensively at the endpoint. #### Summary - The upgrade handshake is an ordinary HTTP exchange; the `101 Switching Protocols` line does not report an outcome, it announces that a different protocol will now be spoken on the connection. - The same forty exchanges are measured with 195 correct and 5 wrong decisions and no round debt in the open regime, while in the tunnel regime all 200 of the 200 decisions are unavailable and forty round debts arise. - The only message the intermediary can read for the life of the connection is the upgrade request, and it gives only two of the five decisions; neither the 101 response nor any of the frames produces a decision. - The frame header runs from two to fourteen bytes and the fields it carries — FIN, opcode, mask, length — are about the transport; cutting bytes does not cost a decision, the field disappearing does. - The two hundred carried decisions turn into work for the endpoints; in return, the number of wrong decisions taken in the tunnel regime is zero. #### Next Step The tunnel moved all two hundred decisions to the endpoints because it cuts the intermediary off from the message entirely. But is changing the connection's shape actually required for the server to send data to the client on its own? The response's body does not have to be written in one go: it can be left open and written to piece by piece, and that stays within HTTP's own rules. The next lesson measures this option — how many decisions the intermediary keeps when it can still read the message, and what being able to read it actually buys. ### Server-Sent Events Source: https://academia.sh/en/courses/application-protocols/server-sent-events What staying HTTP leaves the intermediary: the five decisions the head message carries, the decision point thinning out as event count rises, the wrong readings that a once-taken decision produces when applied to the whole body, and how long a buffering intermediary delays events. The previous lesson counted the tunnel's price: after the upgrade the intermediary loses all two hundred decisions, and all forty of the forty exchanges turn into a round debt. But changing the connection's shape is not actually required for the server to send data to the client on its own. Nothing anywhere requires an HTTP response's body to be written in one go: the body can be left open and written to piece by piece. **Server-Sent Events (SSE)** ties this option to a fixed form. A single response to a single GET request is left open, and the server writes events to it in sequence. This lesson's measure is: because the stream **stays HTTP**, the intermediary can still read the message — so how many decisions are left in its hands, and what does it pay in return for being able to read it? #### The Stream Response The request is ordinary. So is the response; the only things that set it apart are the **media type** and the fact that the body never ends. ```text GET /stream/measurement HTTP/1.1 Host: station.example Accept: text/event-stream HTTP/1.1 200 OK Content-Type: text/event-stream Cache-Control: no-store Transfer-Encoding: chunked event: measurement id: 17 data: {"station": "north-slope", "value": 3} : a comment line produces no event event: measurement id: 18 data: {"station": "north-slope", "value": 5} ``` The body is line-based: field lines build an event, a blank line closes it, and a line starting with a colon is a comment. Because the length cannot be known in advance, the body ends with **chunked transfer** or with the connection closing. The detail of the field format, reconnection based on the `id` field, and the cost the server side pays per connection were measured in the Caching, Queues and Asynchronous Processing course. **What is measured here is the decision the intermediary in between can take from this message.** #### The Decision Point - **OP6.** The forty exchanges carried over a single connection in the previous lesson are carried here in a single stream's body. `event` is the number of exchanges written into a single stream's body; `event = 1` is the request–response shape where every exchange opens its own request. - **OP7.** The intermediary takes its decision from the stream's **head message.** Events written into the body do not carry their own request line or header fields; they are invisible to the intermediary. - **OP8.** The `scope wrong` column is a second reading: it counts how many decisions come out wrong when the decision taken from the head message is applied to **every event** in the body. - **OP9.** The last row is the tunnel regime, where no field of the message can be read at all; it is included here for comparison with the previous lesson's measurement. ```python SEED = 20260809 SAFE = {"GET", "HEAD"} IDEMPOTENT = {"GET", "HEAD", "PUT", "DELETE"} METHOD_POOL = ["GET", "GET", "GET", "HEAD", "POST", "PUT", "DELETE"] PATH_POOL = ["/measurement/north", "/measurement/slope", "/summary", "/register", "/session"] DECISIONS = ("storable", "shareable", "fresh", "repeatable", "redirectable") OPEN = ("method", "path", "host", "private_marker", "validator") def generator(seed: int): d = seed % 2147483646 + 1 def r(n: int) -> int: nonlocal d d = (d * 48271) % 2147483647 return d % n return r def exchanges(count: int = 40) -> list[dict]: r, out = generator(SEED), [] for i in range(count): method, path = METHOD_POOL[r(7)], PATH_POOL[r(5)] private = path == "/session" or r(5) == 0 stale = r(3) == 0 out.append({"no": i + 1, "method": method, "path": path, "private": private, "stale": stale, "private_marker": private and r(4) != 0, "tag_missed": stale and r(6) == 0}) return out def oracle(a: dict) -> dict: return {"storable": a["method"] in SAFE and not a["private"], "shareable": not a["private"], "fresh": not a["stale"], "repeatable": a["method"] in IDEMPOTENT, "redirectable": True} def message(a: dict, visible: tuple) -> dict: full = {"method": a["method"], "path": a["path"], "host": "station.example", "private_marker": a["private_marker"], "validator": a["stale"] and not a["tag_missed"]} return {k: v for k, v in full.items() if k in visible} def infer(i: dict) -> dict: k = {} if "method" in i: k["repeatable"] = i["method"] in IDEMPOTENT if "private_marker" in i: k["shareable"] = not i["private_marker"] if "method" in i: k["storable"] = i["method"] in SAFE and not i["private_marker"] if "validator" in i: k["fresh"] = not i["validator"] if "host" in i: k["redirectable"] = True return k def measure_stream(event: int, visible: tuple = OPEN) -> tuple: out = exchanges() points = correct = wrong = unavailable = extra_round = total_bytes = scope_wrong = 0 for i, a in enumerate(out): m = message(out[i - i % event], visible) # fields of the message that opens the stream head, truth = infer(m), oracle(a) scope_wrong += sum(head[k] != truth[k] for k in head) given = head if i % event == 0 else {} # decision taken only at stream start correct += sum(given[k] == truth[k] for k in given) wrong += sum(given[k] != truth[k] for k in given) unavailable += len(DECISIONS) - len(given) extra_round += 1 if len(given) < len(DECISIONS) else 0 if i % event == 0: points += 1 total_bytes += sum(len(k) + len(str(v)) + 4 for k, v in m.items()) return points, correct, wrong, unavailable, extra_round, total_bytes, scope_wrong fmt = "{:>5s} {:>6d} {:>6d} {:>7d} {:>11d} {:>4d} {:>6d} {:>14d}" print(f"{'event':>5s} {'points':>6s} {'correct':>6s} {'wrong':>7s} " f"{'unavailable':>11s} {'round':>4s} {'bytes':>6s} {'scope wrong':>14s}") for event in (1, 2, 4, 8, 20, 40): print(fmt.format(str(event), *measure_stream(event))) print(fmt.format("-", *measure_stream(40, visible=()))) out = exchanges() head = infer(message(out[0], OPEN)) print() print("under event=40 scope, wrong per decision (across 40 events):") for k in DECISIONS: w = sum(head[k] != oracle(a)[k] for a in out) print(f" {k:<13s} value at stream start {str(head[k]):<5s} wrong {w:2d}/40") ``` ``` event points correct wrong unavailable round bytes scope wrong 1 40 195 5 0 0 3861 5 2 20 97 3 100 20 1909 37 4 10 47 3 150 30 964 45 8 5 23 2 175 35 477 57 20 2 9 1 190 38 198 71 40 1 4 1 195 39 95 73 - 1 0 0 200 40 0 0 under event=40 scope, wrong per decision (across 40 events): storable value at stream start False wrong 13/40 shareable value at stream start True wrong 17/40 fresh value at stream start True wrong 12/40 repeatable value at stream start False wrong 31/40 redirectable value at stream start True wrong 0/40 ``` The first row is the request–response shape, and it is exactly the open regime the course measures: 195 correct, 5 wrong, no unavailable decision, no round debt, 3861 bytes. The last row is the previous lesson's tunnel regime: 0 decisions, 200 unavailable, 40 round debts. The rows in between show what a stream does. When forty events are written into a single body, the intermediary stays at **one** decision point and has **five** decisions in hand. That is five more than the tunnel's zero, and the gap comes from exactly one place: the response that opens the stream **is a readable HTTP response.** In the previous lesson the 101 response gave zero decisions because it carried neither the private flag nor the validator; here the response is an ordinary 200 and it feeds all five of the five decisions. The drop in bytes comes from the same place. What brings 3861 bytes down to 95 is that thirty-nine of the forty messages' header fields are never sent at all. The field is gone, and so is the decision — this runs in the same direction as the previous lesson's result. **Bytes and decisions drop together, because the field is what carries both.** #### The Scope of a Decision The decision point thinning out is not the whole loss. An intermediary does not forget the decision it took at the start of the stream — it applies it to the **whole body.** If it counted the response cacheable, it caches the whole body; if it counted it fresh, it treats the whole body as fresh. The `scope wrong` column measures this. At `event = 1` the head message and the event are the same thing, and the column gives the same five wrong decisions. At `event = 40` a single decision set is applied to forty events, and **73** of the two hundred decisions come out wrong. The decision point has not gone from one to forty; **the decision's scope** has spread from one to forty. The breakdown says where the 73 comes from. `redirectable` is never wrong, because its value is the same on every exchange; a widening scope does not break a decision that does not change. `repeatable` is wrong 31 times: the head message is a POST and its decision came out `False`, yet most of the events in the body belong to safe methods. The remaining three — `storable` 13, `shareable` 17, `fresh` 12 — come from personalization and staleness changing from event to event. The rule is this: **a widening scope only breaks decisions that change from exchange to exchange.** This is the stream's quietest cost. Measured at the decision point, the intermediary has only **one** wrong decision, and that is fewer than the open regime's five. Measured across the same stream's scope, the wrong count rises from five to 73. Given the set's resolution of 1/200 = 0.0050, this gap sits well above the measurable band, and it does not come from a reading error — it comes from **how long the decision is treated as valid.** This is the reason a stream response is closed off from caching. #### Reconnection Is a Decision Point The table's `event` column answers one more question. A stream never stays up forever; the connection breaks, the client reconnects. Reconnecting is a new request, and a new request is **a new decision point.** A stream that breaks once every twenty events is the table's `event = 20` row: two decision points, ten decisions. A stream that breaks every eight events gives five points and twenty-five decisions. What the intermediary sees is inversely proportional to how stable the stream is. One consequence of this is routing. The `redirectable` decision is taken once, at the start of the stream, and **frozen** for the rest of it. In request–response shape the intermediary can route each of the forty exchanges separately; when an upstream resource does not answer, it sends the next request somewhere else. A single stream has no such flexibility: the choice is made once, and the only way to change it is to break the stream. Breaking it also produces a new decision point, so the intermediary can only fix its routing once the client reconnects. The field that makes breaking cheap — the client attaching the last event id it saw to the new request — produces no decision at all in this table. The intermediary reads that field but has nothing to derive from it; the field is a contract between the two ends. This is another instance of the course's fourth reading: **a field existing does not mean the intermediary can derive a decision from it.** #### Buffering Because the intermediary can read the message, it also sees the body, and it makes a choice about what it sees: relay the bytes immediately, or hold them and send in a batch. Batching is an ordinary efficiency rule; on a body that never ends, it delays events. - **OP10.** The intermediary relays no byte from the body until the buffer reaches the threshold. A threshold of 0 means no buffering. Event texts are produced in the specification's line format; the values are the setup. ```python def event_text(no: int, value: int) -> str: """The specification's line format: field lines and the blank line that closes the event.""" return (f"event: measurement\nid: {no}\n" f'data: {{"station": "north-slope", "value": {value}}}\n\n') EVENTS = [event_text(i + 1, i * 3 % 97) for i in range(40)] def buffer(threshold: int) -> tuple: buffered, group, waits = 0, [], [] for i, m in enumerate(EVENTS): buffered += len(m) group.append(i) if buffered >= threshold: waits.extend(i - j for j in group) # how many events each event waited buffered, group = 0, [] average = sum(waits) / len(waits) if waits else 0.0 return len(waits), max(waits, default=0), average, len(group) print(f"event count {len(EVENTS)}, event size " f"{min(len(m) for m in EVENTS)}-{max(len(m) for m in EVENTS)} bytes, " f"body {sum(len(m) for m in EVENTS)} bytes") print() print(f"{'threshold':>9s} {'delivered':>9s} {'max wait':>15s} " f"{'average wait':>17s} {'held at end':>14s}") for threshold in (0, 128, 256, 1024, 4096): delivered, worst, average, held = buffer(threshold) w = str(worst) if delivered else "-" a = f"{average:.2f}" if delivered else "-" print(f"{threshold:9d} {delivered:9d} {w:>15s} {a:>17s} {held:14d}") ``` ``` event count 40, event size 71-73 bytes, body 2904 bytes threshold delivered max wait average wait held at end 0 40 0 0.00 0 128 40 1 0.50 0 256 40 3 1.50 0 1024 30 14 7.00 10 4096 0 - - 40 ``` Two readings apply. In the first three rows, delay grows with the threshold: a 256-byte buffer delays events by 1.5 events on average. The last row is a different thing. The stream produces 2904 bytes of total body across forty events, and a 4096-byte threshold is larger than the whole body; the intermediary relays no byte at all, and until the stream closes it has **never existed** for the client. The critical point is that the intermediary is not doing anything wrong. Batching a response is correct behavior for a body that ends. What it does not know is that this body will not end, and the only field that tells it so is the **media type**. An intermediary that does not recognize the `text/event-stream` value has no other marker at hand: the body length is not declared, the method is GET, the status code is 200, and none of these says "this body will not end." The precaution the server side takes with a heartbeat line closes the same gap from the other end, and it was measured within that course. #### One Direction The stream is one-way: server to client. When the client has something to tell the server, it cannot write into the stream — it opens an ordinary request instead. This is a detail that does not look like a loss in the measurement table, but its consequence is direct: **every message coming from the client is a decision point again.** On the stream side the decision point has dropped to one; every request the client writes to the server is a readable exchange like the table's first row, and it brings its own decision point back. The tunnel does not even have this. Every two-way message there is in the same frame format, and none of them produces a decision. The gap also shows what the choice is made on: if two-way communication is genuinely needed, the tunnel; if only server-side streaming is needed, SSE. The cost of direction is measurable in decisions. #### Summary - Server-Sent Events stays inside HTTP: the request is an ordinary GET, the response is an ordinary 200, and the body is left open and written to with events. - When forty events are carried in a single stream, the intermediary's decision point drops to one and it keeps five decisions in hand; in the tunnel regime this count is zero, and the gap comes from the response that opens the stream being readable. - The decision taken from the head message is applied to the whole body; the wrong count measured at the decision point is 1, while the wrong count measured across the scope is 73 of two hundred decisions. - Bytes drop from 3861 to 95 while decisions drop from 200 to 5; what carries both is the field. - A buffering intermediary delays events by 1.5 events on average at a 256-byte threshold; against a 2904-byte body, a 4096-byte threshold delivers no event at all, and the only field that says the body will not end is the media type. #### Next Step Both the stream and the tunnel work by keeping the connection **open**: one never finishes the body, the other changes the protocol. Both extend the connection's lifetime to free the server's hand. But how far can one get without doing either — without adding a single new field to the protocol — just by **asking more often**? The next lesson measures this: short polling, long polling and server streaming are set side by side over the same event set, and what divides them is not the event, it is the **decision**. ### Long and Short Polling Source: https://academia.sh/en/courses/application-protocols/long-and-short-polling Real-time behavior faked by buying rounds: polling adding not a single new field to the protocol, the round cost per decision staying unchanged across four approaches, and the sum of spent rounds and round debt staying fixed. The previous two lessons measured two ways of keeping the connection **open.** The tunnel changed the protocol, the stream never finished the body; both extended the connection's lifetime so the server could speak on its own. This lesson's question is: without doing either, without adding a single new field to the protocol, just by **asking more often**, how far can one get? **Polling** is the client sending requests at regular intervals to learn the state. In **short polling** the server answers right away; if it has nothing new to give, it says so. In **long polling** the server delays the response; it holds the request until either a change happens or the hold time runs out. This lesson's measure is what each pays in **rounds per decision.** One caution: the `ping` control frame from the previous lesson and the polling in this lesson are separate things — one is a frame that confirms the connection is up, the other is a round the application takes to learn the state. #### Same Request, Different Timing The difference between the two polling forms **has no counterpart in the message.** ```text # short polling — the server answers right away GET /measurement/north HTTP/1.1 Host: station.example If-None-Match: "s-118" HTTP/1.1 304 Not Modified ETag: "s-118" # long polling — the request line and headers are the same, the response is delayed GET /measurement/north HTTP/1.1 Host: station.example If-None-Match: "s-118" HTTP/1.1 200 OK ETag: "s-119" Content-Type: application/json {"station": "north-slope", "value": 5} ``` The two requests are identical. There is no new method, no new header field, no new status code. The conditional-request field is not polling's invention either; it is HTTP's own field, built in this course's HTTP family topic. The whole difference is **when the server writes**, and time is not something written in the message. The cost polling puts on the server — request count, empty-response ratio, held connections — was measured in the Caching, Queues and Asynchronous Processing course. **What divides here is the decision, not the event, and what is measured is the intermediary's account in between.** #### The Measurement - **OP11.** The station's state is watched over forty ticks. Whether the state changes on a given tick is the setup's oracle; the state changes in twelve of the forty ticks. - **OP12.** A **round** is one round trip in which the client sends a request and gets a response. An empty response does not make a round cheaper. - **OP13.** Short polling asks once every `interval` ticks. Long polling holds until a change happens; if the `hold` tick runs out, it gives an empty response and the client reconnects. - **OP14.** Server streaming and the tunnel spend a single round; after that there is no new message left for the intermediary to read. - **OP15.** Every five unavailable decisions is one **round debt**: if the intermediary wants that exchange's decision, it has to go back to the server. ```python SEED = 20260809 SAFE = {"GET", "HEAD"} IDEMPOTENT = {"GET", "HEAD", "PUT", "DELETE"} METHOD_POOL = ["GET", "GET", "GET", "HEAD", "POST", "PUT", "DELETE"] PATH_POOL = ["/measurement/north", "/measurement/slope", "/summary", "/register", "/session"] DECISIONS = ("storable", "shareable", "fresh", "repeatable", "redirectable") OPEN = ("method", "path", "host", "private_marker", "validator") def generator(seed: int): d = seed % 2147483646 + 1 def r(n: int) -> int: nonlocal d d = (d * 48271) % 2147483647 return d % n return r def exchanges(count: int = 40) -> list[dict]: r, out = generator(SEED), [] for i in range(count): method, path = METHOD_POOL[r(7)], PATH_POOL[r(5)] private = path == "/session" or r(5) == 0 stale = r(3) == 0 out.append({"no": i + 1, "method": method, "path": path, "private": private, "stale": stale, "private_marker": private and r(4) != 0, "tag_missed": stale and r(6) == 0}) return out def oracle(a: dict) -> dict: return {"storable": a["method"] in SAFE and not a["private"], "shareable": not a["private"], "fresh": not a["stale"], "repeatable": a["method"] in IDEMPOTENT, "redirectable": True} def message(a: dict, visible: tuple) -> dict: full = {"method": a["method"], "path": a["path"], "host": "station.example", "private_marker": a["private_marker"], "validator": a["stale"] and not a["tag_missed"]} return {k: v for k, v in full.items() if k in visible} def infer(i: dict) -> dict: k = {} if "method" in i: k["repeatable"] = i["method"] in IDEMPOTENT if "private_marker" in i: k["shareable"] = not i["private_marker"] if "method" in i: k["storable"] = i["method"] in SAFE and not i["private_marker"] if "validator" in i: k["fresh"] = not i["validator"] if "host" in i: k["redirectable"] = True return k def short(interval: int, out: list[dict]) -> list[int]: """Asked once every `interval` ticks; the response returns the state at that tick.""" return list(range(interval - 1, len(out), interval)) def long(hold: int, out: list[dict]) -> list[int]: """The server holds until a change happens; a `hold` tick fills up with an empty response.""" rounds, t = [], 0 while t < len(out): end = min(t + hold - 1, len(out) - 1) for u in range(t, end + 1): if out[u]["stale"]: end = u break rounds.append(end) t = end + 1 return rounds def measure(rounds: list[int], visible: tuple = OPEN) -> tuple: out = exchanges() correct = wrong = total_bytes = 0 fields = set() for t in rounds: m = message(out[t], visible) given, truth = infer(m), oracle(out[t]) correct += sum(given[k] == truth[k] for k in given) wrong += sum(given[k] != truth[k] for k in given) total_bytes += sum(len(k) + len(str(v)) + 4 for k, v in m.items()) fields |= set(m) taken = correct + wrong ratio = len(rounds) / taken if taken else float("nan") return (len(rounds), len(fields), taken, correct, wrong, 200 - taken, ratio, total_bytes) OUT = exchanges() print(f"tick {len(OUT)}, decision set {len(OUT) * len(DECISIONS)}, " f"changes {sum(a['stale'] for a in OUT)}") print() fmt = "{:<18s} {:>4d} {:>5d} {:>7d} {:>6d} {:>7d} {:>11d} {:>10.3f} {:>6d}" print(f"{'approach':<18s} {'round':>4s} {'fields':>5s} {'taken':>7s} " f"{'correct':>6s} {'wrong':>7s} {'unavailable':>11s} {'round/decision':>10s} " f"{'bytes':>6s}") APPROACHES = [("short polling (1)", short(1, OUT), OPEN), ("short polling (4)", short(4, OUT), OPEN), ("long polling (5)", long(5, OUT), OPEN), ("server stream", [0], OPEN), ("tunnel", [0], ())] for name, rounds, visible in APPROACHES[:-1]: print(fmt.format(name, *measure(rounds, visible))) n, fields, taken, c, w, e, ratio, b = measure([0], visible=()) print(f"{'tunnel':<18s} {n:4d} {fields:5d} {taken:7d} {c:6d} {w:7d} {e:11d} " f"{'-':>10s} {b:6d}") print() print(f"{'approach':<18s} {'spent round':>12s} {'round debt':>9s} " f"{'total':>6s}") for name, rounds, visible in APPROACHES: m = measure(rounds, visible) spent, debt = m[0], m[5] // len(DECISIONS) print(f"{name:<18s} {spent:12d} {debt:9d} {spent + debt:6d}") ``` ``` tick 40, decision set 200, changes 12 approach round fields taken correct wrong unavailable round/decision bytes short polling (1) 40 5 200 195 5 0 0.200 3861 short polling (4) 10 5 50 49 1 150 0.200 992 long polling (5) 15 5 75 72 3 125 0.200 1445 server stream 1 5 5 4 1 195 0.200 95 tunnel 1 0 0 0 0 200 - 0 approach spent round round debt total short polling (1) 40 0 40 short polling (4) 10 30 40 long polling (5) 15 25 40 server stream 1 39 40 tunnel 1 40 41 ``` The first row is the course's open regime itself: forty rounds, 195 correct, 5 wrong, no unavailable decision, 3861 bytes. Pushing the interval to four cuts the round count to ten and drops decisions taken to fifty. Long polling spends twelve rounds on the twelve changes and three more on the hold time running out: fifteen total. #### The Round's Constant The `fields` column is **five** in all four HTTP approaches. Polling adds no field to the protocol — not the short form, not the long form. Server streaming adds none either. The only thing that gets added is **rounds**, and how many of them there are is not written in the message. The `round/decision` column is the direct result: **0.200** in all four approaches. This number is 1/5, and it is no coincidence. A readable exchange carries five fields, five fields give five decisions; a round therefore always buys exactly five decisions. Changing polling's frequency cannot **move** this ratio, because both its numerator and its denominator come from the same field set. This produces the course's second claim in its third form. Adding a field bought no decision; hiding a field took a decision away; now it turns out that **adding a round does not change the per-decision cost either.** Polling fakes real-time behavior while taking nothing from the protocol; it only does the same operation more often. The tunnel row sits outside this table and no ratio can be computed for it: when the field count is zero, so is the denominator. If a round buys zero decisions, buying rounds has no meaning either. #### Round Accounting The second table splits the same forty exchanges into two kinds of round. **Spent round** is a round trip the client actually made. **Round debt** is a round trip the intermediary will have to make back to the server for decisions it could not take. In all four HTTP approaches the total is **forty** and it does not change. This is the plainest form of the course's axis. The decisions on forty exchanges have to come from somewhere: either the client pays the round up front and the decision arrives in the message, or the round is written as debt and the decision is not taken. Short polling drives the debt to zero and pays all forty rounds up front. Server streaming pays one round and leaves thirty-nine in debt. Every approach in between is a point on this same line. The tunnel row gives forty-one. The extra round is the upgrade handshake itself: one round is spent and no decision comes back for it. All forty of the forty exchanges stay in debt. The price of hiding sits exactly in the one row that spills past the table. #### What the Held Request Corresponds to at the Intermediary Long polling has a side effect for the intermediary. While the server is holding the request, no byte flows over the connection; what the intermediary sees is a request whose response is delayed. Nothing in the intermediary's hands says this wait is **deliberate** — the request line is an ordinary GET, and none of the headers say "I'm going to keep you waiting." The idle timeout from the previous lesson, meanwhile, is an intermediary's ordinary rule, and it cuts the held request without knowing it was meant to be held. - **OP16.** The intermediary cuts a connection with no byte flowing on it once it passes the idle timeout threshold; the client reopens the cut round. The table's `idle cutoff` column gives this threshold in ticks. ```python SEED = 20260809 PATH_POOL = ["/measurement/north", "/measurement/slope", "/summary", "/register", "/session"] def generator(seed: int): d = seed % 2147483646 + 1 def r(n: int) -> int: nonlocal d d = (d * 48271) % 2147483647 return d % n return r def changes(count: int = 40) -> list[bool]: """Only the `stale` flag is taken from the previous block's generator; call order is preserved.""" r, out = generator(SEED), [] for _ in range(count): r(7) # method selection path = PATH_POOL[r(5)] private = path == "/session" or r(5) == 0 stale = r(3) == 0 if private: r(4) # private flag if stale: r(6) # validator resolution out.append(stale) return out def long_rounds(hold: int, changed: list[bool]) -> list[tuple]: """Each round is a (start, end) tick pair; the end is either a change or the hold limit.""" rounds, t = [], 0 while t < len(changed): end = min(t + hold - 1, len(changed) - 1) for u in range(t, end + 1): if changed[u]: end = u break rounds.append((t, end)) t = end + 1 return rounds ROUNDS = long_rounds(5, changes()) DURATIONS = [e - b + 1 for b, e in ROUNDS] print(f"long polling (hold 5): {len(ROUNDS)} rounds, durations {DURATIONS}") print() print(f"{'idle cutoff':>11s} {'cut rounds':>11s} {'total rounds':>13s} " f"{'decisions taken':>16s} {'round/decision':>15s}") for threshold in (1, 2, 3, 5, 10): cut = sum(1 for d in DURATIONS if d > threshold) total = len(ROUNDS) + cut print(f"{threshold:11d} {cut:11d} {total:13d} {total * 5:16d} " f"{total / (total * 5):15.3f}") ``` ``` long polling (hold 5): 15 rounds, durations [2, 5, 5, 3, 5, 1, 1, 2, 1, 3, 4, 4, 1, 1, 2] idle cutoff cut rounds total rounds decisions taken round/decision 1 10 25 125 0.200 2 7 22 110 0.200 3 5 20 100 0.200 5 0 15 75 0.200 10 0 15 75 0.200 ``` The `decisions taken` column is the round count multiplied by five; the previous measurement already showed that every readable exchange gives exactly five decisions. A one-tick idle cutoff cuts ten of the fifteen rounds and drives the total up to twenty-five. The round long polling gained is thereby taken back: a design that would get by on fifteen rounds spends twenty-five because of the intermediary's threshold. Pulling the hold time below the threshold fixes it — at a five-tick threshold no round gets cut — but that means the application is choosing **its own design according to a setting on the path.** The `round/decision` column stops at 0.200 here too. A cut round is a readable exchange as well, and it too gives five decisions. Cutting does not change the **ratio** of decisions the intermediary takes; it only raises the round the client pays. #### What a Round Buys It is also worth saying what a round buys where it does not buy a decision: **freshness.** In short polling, when the interval is one tick, the client learns every change on the very tick it happens. At a four-tick interval it learns, in the worst case, three ticks late. Long polling closes this delay without spending a round — because the server waits for the change, the response is written at the exact moment of the change — and gives the freshness of forty rounds for fifteen. This is the real distinction among the three approaches, and it never shows up in the decision table. The decision table is the intermediary's account; freshness is the client's account. A protocol choice weighs both at once: how many decisions the intermediary can take, and how much new information the client gets for how many rounds. The measurement band's lower bound holds here too; on a set of two hundred decisions the smallest measurable difference is 1/200 = 0.0050, and every difference in the table sits well above it. #### Summary - Short and long polling's requests are identical to each other; the difference is when the server writes the response, and time is not a field written in the message. - In all four HTTP approaches the number of fields the intermediary reads is five; polling adds not a single new field to the protocol. - The round cost per decision is 0.200 in all four approaches, because a readable exchange always gives exactly five decisions; changing the frequency cannot move this ratio. - The sum of spent rounds and round debt is forty in all four HTTP approaches: short polling 40 + 0, long polling 15 + 25, server stream 1 + 39. - The tunnel pushes the total to forty-one; the extra round is the upgrade handshake, for which no decision comes back. - What a round buys is not a decision but freshness, and freshness never shows up in the intermediary's table. #### Next Step All three approaches sat between the **two ends** of a single application, and all three used the same transport envelope: the same host name, the same connection, the same five fields. The intermediary's account could therefore be kept in a single ledger — which round belonged to which exchange was always known. Some protocols, though, do not even use a single connection. The next lesson measures a protocol family whose messages spread across more than one connection, and asks: when the intermediary cannot match the round it sees to the round it cannot see, what is left in its ledger? ### File Transfer Protocols Source: https://academia.sh/en/courses/application-protocols/file-transfer-protocols The split between the control channel and the data channel: which decision the intermediary loses and which it gets wrong when it cannot match the two channels, and how the cost of splitting is separated from the cost of hiding. The previous three lessons set the three roads to real-time behavior side by side and measured the round per decision. All three shared one thing: each sat between **the two ends of a single application**, and each used the same **transport envelope**. The intermediary was looking at a single connection; what it saw, it either saw whole or not at all. Some protocols do not work this way. A file transfer can split a single logical job across **two separate connections**: commands on one, the file's bytes on the other. This lesson's question is: if the intermediary sees both but has no field telling it **which belongs to which**, what does it lose? The loss has nothing to do with hiding; this lesson counts those two costs separately. - **OP41.** The setup is M04/K03's shared setup: the `station.example` measurement station, and an intermediary in between that does not know the application. The intermediary looks only at what is written in the message. - **OP42.** The set measured is **40 transfers** and **5 decisions** per transfer: `is_upload`, `retryable`, `is_measurement_file`, `resumable`, `completed`. Total 200 decisions; the smallest measurable difference in this set is $1/200 = 0{,}0050$. - **OP43.** The oracle — the truth itself — is known because we produced the setup ourselves, and it does not change from regime to regime. The regime only changes the fields the intermediary **sees**. - **OP44.** Four regimes are set up: two channels open, data channel wrapped, single channel open, single channel wrapped. The transfers themselves are the same across all four regimes. - **OP45.** The use of transfer tools is not repeated here. Remote shell, key transport and synchronization procedure were measured in the Linux Network Administration and Troubleshooting course; the measure here is **channel separation**, not the tool. #### The Control Channel and the Data Channel The classic form of file transfer defines two channels. The **control channel** carries commands and reply codes; it stays open for the whole session. The **data channel** carries only the bytes of the file or the directory listing; it opens for each transfer and closes when the transfer ends. FTP is the specification that makes this split. ```text control channel — between the client and station.example, open for the whole session -> USER measurement <- 331 password required -> PASS ******** <- 230 session opened -> TYPE I <- 200 type binary -> PASV <- 227 passive mode (a1,a2,a3,a4,p1,p2) -> SIZE /measurement/north.csv <- 213 9216 -> RETR /measurement/north.csv <- 150 opening data channel <- 226 transfer complete data channel — a separate connection, for the duration of the transfer 9216 bytes flow. No command, no path, no session marker; the connection closes when it ends. ``` This dump is not run; it is the taught form. Two points should be read. First, **where the data channel will open is negotiated in the control channel's body**: with `PORT` the client announces where it is listening itself, with `PASV` the server announces where it has opened its own end. Second, the bytes flowing on the data channel carry **no field that says which command they belong to.** Two designs sit at the opposite end. SFTP opens a single connection, wraps that connection in an envelope, and **frames** both commands and bytes over the same connection; every request carries a request marker and the response comes back with that marker. FTPS keeps the two-channel layout but wraps each channel in its own envelope — and "each" is the key word here. #### Resume and Directory Listing Two of the five measured decisions come from these two details, so they need to be set up first. **Resume.** A transfer cut short partway can be completed in two ways: from the start, or from where it left off. Resuming from where it left off is done by the client declaring a starting offset and repeating the command; in the specification the command for this is `REST`. Whether the server supports resuming is **a capability declared on the control channel**, and nothing verifies the declaration. This decision has a concrete cost for the intermediary: if resuming is supported, a cut-short transfer only costs the remaining bytes; if not, the whole file flows again. In the measurement every transfer is a single attempt; resume's effect on **byte accounting** should also be noted separately, because in a resumed transfer the byte count flowing on the data channel is smaller than the declared size, and an intermediary that looks at the size comparison mistakes a successful transfer for an incomplete one. **Directory listing.** The format of the directory listing the `LIST` command returns on the data channel is not fixed in the specification; it is designed for a human to read. The result is the sharpest form of the course's rule: **the field exists, but it has no meaning.** The intermediary sees the bytes, can even split the lines, but has no contract it can rely on for which column is size and which is name. The specification closes this gap later with a listing command meant to be read by machine — `MLSD`; the way it closes the gap is not a new byte, it is **fields whose name and meaning are fixed.** In the measurement, `LIST` accordingly sits in the same class as `RETR` on the `retryable` side: neither changes the station's state. #### The Fields in the Intermediary's Hands The intermediary reads the command, the path, the size the server declares, and the resume support from the control channel. From the data channel it reads only a byte count. To combine these two observations it has exactly one thing at hand: **time proximity.** It counts the connection opened right after a command as that command's connection. Proximity is not a substitute for a marker. If another session is also connected to the same station, the connection the intermediary counts as adjacent may belong to someone else. The measurement counts this. The difference between active and passive mode is, from the intermediary's side, a matter of **direction**: in active mode the server opens the data connection, in passive mode the client does. The intermediary sees both, but what it sees is a connection setup, not a command. Who opens the connection and **which way the bytes flow** are two separate things, and the second cannot be derived from the first. The direction of the bytes is read only from the command on the control channel: `RETR` downloads, `STOR` uploads. This is the only source of the `is_upload` decision measured here, and when the command is invisible, that decision goes too — this is why the fourth regime's zero is zero. ```python SEED = 20260809 COMMAND_POOL = ["RETR", "RETR", "STOR", "LIST", "RETR", "STOR", "RETR"] PATH_POOL = ["/measurement/north.csv", "/measurement/slope.csv", "/summary.csv", "/register.log", "/measurement/daily.csv"] DECISIONS = ("is_upload", "retryable", "is_measurement_file", "resumable", "completed") def generator(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def transfers(count=40, seed=SEED): r, out = generator(seed), [] for i in range(count): command, path = COMMAND_POOL[r(7)], PATH_POOL[r(5)] declared = 1024 * (1 + r(9)) cut = r(5) == 0 resumable = r(3) != 0 out.append({ "no": i + 1, "command": command, "path": path, "declared": declared, "received": declared - 512 if cut else declared, "resumable": resumable, # truth "resume_flag": resumable and r(7) != 0, # declaration "neighbor_session": r(5) == 0, # adjacent connection belongs to someone else "neighbor_bytes": 1024 * (1 + r(9)), }) return out def oracle(a): return { "is_upload": a["command"] == "STOR", "retryable": a["command"] in ("RETR", "LIST"), "is_measurement_file": a["path"].startswith("/measurement/"), "resumable": a["resumable"], "completed": a["received"] == a["declared"], } def message(a, visible): full = { "command": a["command"], "path": a["path"], "host": "station.example", "resume_flag": a["resume_flag"], "declared": a["declared"], "adjacent_bytes": a["neighbor_bytes"] if a["neighbor_session"] else a["received"], "channel_bytes": a["received"], "session_marker": a["no"], } return {k: v for k, v in full.items() if k in visible} def infer(i): k = {} if "command" in i: k["is_upload"] = i["command"] == "STOR" k["retryable"] = i["command"] in ("RETR", "LIST") if "path" in i: k["is_measurement_file"] = i["path"].startswith("/measurement/") if "resume_flag" in i: k["resumable"] = i["resume_flag"] if "declared" in i: if "session_marker" in i: k["completed"] = i["channel_bytes"] == i["declared"] elif "adjacent_bytes" in i: k["completed"] = i["adjacent_bytes"] == i["declared"] return k CONTROL = ("command", "path", "host", "resume_flag", "declared") VISIBLE = { "two channels open": CONTROL + ("adjacent_bytes",), "data channel wrapped": CONTROL, "single channel open": CONTROL + ("channel_bytes", "session_marker"), "single channel wrapped": ("host",), } def measure(name): correct = wrong = unavailable = extra_round = 0 breakdown = {} for a in transfers(): truth, given = oracle(a), infer(message(a, VISIBLE[name])) missing = False for decision in DECISIONS: if decision not in given: unavailable += 1 missing = True elif given[decision] == truth[decision]: correct += 1 else: wrong += 1 breakdown[decision] = breakdown.get(decision, 0) + 1 extra_round += 1 if missing else 0 return correct, wrong, unavailable, extra_round, breakdown print(f"{'regime':<23s} {'correct':>7s} {'wrong':>6s} {'unavailable':>11s} " f"{'round':>5s} wrong breakdown") for name in VISIBLE: c, w, u, e, dk = measure(name) print(f"{name:<23s} {c:7d} {w:6d} {u:11d} {e:5d} {dk}") a = transfers() print() print(f"transfers {len(a)}, decisions {len(a) * len(DECISIONS)}, " f"cut short {sum(x['received'] != x['declared'] for x in a)}, " f"left unflagged {sum(x['resumable'] and not x['resume_flag'] for x in a)}, " f"neighbor session {sum(x['neighbor_session'] for x in a)}") ``` ``` regime correct wrong unavailable round wrong breakdown two channels open 192 8 0 0 {'completed': 3, 'resumable': 5} data channel wrapped 155 5 40 40 {'resumable': 5} single channel open 195 5 0 0 {'resumable': 5} single channel wrapped 0 0 200 40 {} transfers 40, decisions 200, cut short 7, left unflagged 5, neighbor session 6 ``` #### The Cost of Splitting Is Not the Cost of Hiding The first row and the third row see **the same amount of fields**: the same command, the same path, the same declared size, the same byte count. Nothing is hidden. Yet the two-channel regime takes 192 correct decisions while the single-channel regime takes 195. The **3 decisions** between them come from **splitting alone, and only splitting**: on a single channel the bytes are framed and carry a session marker, so matching is correct by construction; on two channels, the intermediary has mistaken a neighboring session's connection for this command's connection. 3/200 = 0.015; three times the set's resolution, so a measured difference. There is a subtlety in reading the number. A neighboring session steps in on **6 transfers**, but the wrong decision only comes out of **3** of them. In the remaining three, the transfer itself is already cut short: the intermediary says "not complete," the oracle also says "not complete," and the decision comes out correct. **The intermediary gave the right answer for the wrong reason.** The measurement counts this as correct, because what is measured is the decision; but from a design standpoint this is not a guarantee, it is two errors covering for each other. The second row says something entirely different. When the data channel is wrapped, the intermediary cannot see the byte count at all, **never** takes the `completed` decision, and in forty of the forty transfers it owes the server one more round. A misunderstanding should be closed here: a wrapped connection still carries a byte count on the wire, but that number is not the file's size — the envelope's record headers and padding are inside that total too. The intermediary's number and the declared size on the control channel are **not even in the same unit**; this is why the decision does not come out wrong, it does not come out at all. In return, **3 wrong decisions disappear**: the remaining 5 wrong ones all come from another source. This is this lesson's form of the course's third claim — **fewer decisions, but fewer wrong ones too.** A wrapper is not a defect, it is a choice, and the choice's name is: the round debt is paid, the wrong match is not bought. The source of the remaining 5 wrong decisions is neither the split nor the wrapper. In five transfers the server genuinely supports resuming but **has not declared it** on the control channel. The intermediary looks at the declaration, says "not resumable," and is wrong. This is the course's fourth reading, repeated in every lesson: **nothing verifies what is written in the message.** If a field is missing, the intermediary does not know; if it is wrong, it knows wrong. This same five is the same five in three of the four regimes — splitting and hiding neither raise nor lower it. #### Why the Insecure Option Is Insecure The base specification carries both the control channel and the data channel with **no envelope at all.** The result is: every device standing along the path can read the credential passed at session opening, the requested file paths, the declared sizes, and the file's own bytes. This is not a defect, it is the assumption the specification was written under — the path itself was assumed trustworthy. When the assumption does not hold, the protocol has no field to close it. Splitting adds a second point here: **the two channels are protected separately.** In a setup where the control channel is wrapped and the data channel is left open, the client assumes its session is protected, while the file's bytes are unprotected. The measurement's second row is the mirror image of this: there the data channel is wrapped and the control channel is open; the intermediary keeps reading the path and the size. In a two-channel design, the question "is it protected" has **no single answer** — it has one answer per channel. The third point is structural. Because the data channel's address is negotiated in the control channel's body, an intermediary translating the address has to **read and rewrite** that body. When the control channel is wrapped, it cannot do this; the transfer either cannot be set up, or the configuration is forced to keep a whole port range permanently open. Hiding and the middle device's ability to do its job stand in direct conflict here. **Narrowing.** All three of the surfaces counted point the same way: **reduce the channel count to one.** When a single connection is wrapped in an envelope, the credential, the path, the size and the file's bytes are all protected by one decision; framing settles the matching by construction, so the three wrong decisions never arise; and because there is no address negotiation in the body, the middle device does not need to rewrite the body. SFTP goes this way. If the two-channel layout has to be kept, the narrowing is **asking for the protection of both channels separately and explicitly**; one being wrapped says nothing about the other. The measurement's last row also writes down the price: in the single-channel-wrapped regime the intermediary loses all 200 decisions. Its caching, retrying and size checking are absent in this regime; that work returns to the endpoints. #### Summary - Classic file transfer splits one job across two connections: the control channel carries commands and reply codes, the data channel carries only the bytes; nothing on the bytes says which command they belong to. - On the set of 40 transfers and 200 decisions, the two-channel open regime takes 192 correct decisions and the single-channel open regime takes 195; the 3 decisions between them come **only from splitting**, not from hiding. - A neighboring session steps in on 6 transfers but the wrong decision only arises in 3; in the remaining three the transfer is already cut short, so the intermediary gives the right answer for the wrong reason. - When the data channel is wrapped, the `completed` decision cannot be taken 40 times and 40 round debts arise; in return, 3 wrong decisions disappear — fewer decisions, but fewer wrong ones. - The 5 wrong decisions are the same across all three regimes and come from the server not declaring its resume support; neither splitting nor hiding repairs a missing declaration. - In the unwrapped base specification the credential, the path, the size and the file's bytes can all be read along the path; because the two channels are protected separately, the question "is it protected" has one answer per channel, and the narrowing is to reduce the channel count to one. #### Next Step What was split in this lesson was two **channels**: one carried what was requested, the other what arrived. When the intermediary could not match them, it got a decision wrong. In the next lesson what is split will be two **names**. The fields that say where a piece of mail is to be delivered and the fields that show the user who it came from sit in the same message, do not verify each other, and delivery is routed by **looking at the envelope**. The name the intermediary reads for routing and the name the human sees on screen come from two separate fields; the next lesson measures why this split exists and which control closes it. ### Mail Protocols Source: https://academia.sh/en/courses/application-protocols/mail-protocols The split between submission and retrieval, and mail's two names: the mail envelope that routes delivery and the header shown to the user are separate fields, they do not verify each other, and the intermediary looks at the envelope. The previous lesson measured a protocol that splits one job across two **channels**: commands in one, bytes in the other, and the intermediary got a decision wrong when it could not match the two. The loss came from the split, not from hiding. Mail goes one step further. Here what splits is not the channel but the **name**. The fields that say where a piece of mail is to be delivered and the fields that show the user who it came from sit inside the same message, in two separate layers. The layer that routes delivery is the **mail envelope**, and the intermediary looks at it. The name the user sees on screen comes from the header and plays no part in delivery. Neither verifies the other. This lesson's question is **why** this split exists and which control closes it. - **OP46.** The setup is M04/K03's shared setup: the `station.example` measurement station produces alert mail, the domains `example.test` and `list.example` stand for the recipient side and the list side, and an intermediary that does not know the application sits in between. - **OP47.** **The mail envelope and the transport envelope are two separate concepts.** The transport envelope is the layer built in the Secure Transport topic that blinds the intermediary. The mail envelope, in contrast, is the protocol's own fields; it hides nothing, it only routes delivery. What is measured in this lesson is the mail envelope. - **OP48.** The set measured is **40 messages** and **5 decisions** per message: `delivery_path_exists`, `sender_authorized`, `names_aligned`, `has_bcc_recipient`, `displayed_name_verified`. Total 200 decisions; the smallest measurable difference is $1/200 = 0{,}0050$. - **OP49.** Messages are of four kinds: direct, proxied (sent by another domain on the station's behalf), list, and forwarded. **All four are legitimate**; there is no forged message anywhere in the setup. The measurement's purpose is not to count forgeries, it is to count what **legitimate variety** does to controls. - **OP50.** Publishing a domain policy is the domain's own choice: `station.example` and `list.example` publish one, `example.test` does not. The oracle — the truth itself — is known because we produced the setup ourselves and does not change from regime to regime. #### Submission and Retrieval Mail is carried by two separate protocol families, and they face opposite directions. **Submission** is the road that **pushes** the message from the client to the recipient's mail store. SMTP is this road's specification; the same specification is used both for the client's first submission and for the transfer between relays afterward. **Retrieval**, in turn, is the road the user **pulls** a message from once it has reached the store; IMAP and POP3 are this direction's specifications. IMAP keeps the message on the server and the user reads the same view from many places along with the folder structure; POP3 is built on downloading the message and dropping it from the store. The consequence for this lesson is: **the intermediary only stands on the submission road.** On the retrieval road there is no decision-making point between the user and their own store. A decision not taken during submission cannot be taken back during retrieval — the message has already been delivered. The round's debt is written into a mailbox here. The two retrieval specifications also diverge in round behavior. In POP3 the state lives in the client: message numbers inside a session are specific to that session, and once messages are downloaded and dropped from the store, a second client cannot see the same mailbox. In IMAP the state lives on the server; every message carries a persistent identifier that does not change from session to session, along with flags such as read, replied, deleted, so multiple clients share the same view. The price is this: every query on shared state is a round trip. The previous three lessons' polling question comes up on the retrieval side too, in exactly the same shape, and IMAP's answer belongs to the same family — the `IDLE` command holds the connection open, the server reports the change itself, and the client wastes no round. Short polling's round cost per decision holds here too. #### The Mail Envelope and the Header The two layers of the submission conversation sit side by side below. ```text submission conversation — between the client and the delivering host -> EHLO station.example <- 250 hello -> MAIL FROM: mail envelope: return path <- 250 sender accepted -> RCPT TO: mail envelope: delivery address <- 250 recipient accepted -> RCPT TO: mail envelope: second recipient <- 250 recipient accepted -> DATA <- 354 send the message, end with a single dot From: North Slope Station header To: Nobet header Subject: threshold exceeded header Date: ... The measurement exceeded the threshold. . <- 250 message queued ``` This dump is not run; it is the taught form. Four points should be read. First, delivery is done **only** by looking at the `RCPT TO` lines. The `To:` header plays no part in delivery; a delivery that does not read it still works correctly. Second, the message above has two envelope recipients but the header names only one — `arsiv@example.test` never enters the header at all. **A blind carbon copy is not a header field, it is an envelope recipient**; there is no corresponding line for it in the header, and there should not be. Third, `MAIL FROM` and `From:` are two separate addresses. The first is where an undeliverable message will bounce back to, the second is the name to be shown to the user. In a list message, the return path belongs to the list and the displayed name belongs to the original author; the two being different is not a protocol defect, it is the design. A fourth point shows up for the first time in this course: **the intermediary here does not only read, it writes too.** Every host that relays the message adds a `Received:` line to the top of the header, and the accumulating lines record, in reverse order, the path the message took. This is the only place in the course where the intermediary puts its own declaration into the message. The record has a limit worth noting for this lesson: `Received:` lines are **a self-declaration by the hosts that carried the message**, and nothing verifies the lines below the top one. A trace existing does not mean the trace is correct — the course's fourth reading holds here exactly the same way. #### The Measurement The intermediary is measured at three levels of visibility: seeing only the mail envelope, also reading the headers, and also having received the key that signs the header. ```python SEED = 20260809 TYPES = ["direct", "direct", "direct", "proxied", "list", "forwarded", "direct"] ENVELOPE_DOMAIN = {"direct": "station.example", "proxied": "example.test", "list": "list.example", "forwarded": "example.test"} POLICY = {"station.example": True, "list.example": True, "example.test": False} DECISIONS = ("delivery_path_exists", "sender_authorized", "names_aligned", "has_bcc_recipient", "displayed_name_verified") def generator(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def messages(count=40, seed=SEED): r, out = generator(seed), [] for i in range(count): kind = TYPES[r(7)] envelope_recipients = 1 + r(5) out.append({ "no": i + 1, "type": kind, "envelope_domain": ENVELOPE_DOMAIN[kind], "header_domain": "station.example", "envelope_recipients": envelope_recipients, "header_recipients": envelope_recipients - 1 if r(5) == 0 else envelope_recipients, "forwarded": kind in ("list", "forwarded"), "source_authorized": r(9) != 0, # truth: the exit host is actually authorized "signature_present": r(7) != 0, "altered": r(9) == 0, # truth: the message was altered in transit "path_exists": r(11) != 0, }) return out def oracle(m): return { "delivery_path_exists": m["path_exists"], "sender_authorized": m["source_authorized"], "names_aligned": m["envelope_domain"] == m["header_domain"], "has_bcc_recipient": m["envelope_recipients"] > m["header_recipients"], "displayed_name_verified": m["signature_present"] and not m["altered"], } def message(m, visible): full = { "envelope_domain": m["envelope_domain"], "envelope_recipients": m["envelope_recipients"], "path_exists": m["path_exists"], "deliverer_authorized": m["source_authorized"] and not m["forwarded"], "header_domain": m["header_domain"], "header_recipients": m["header_recipients"], "signature_present": m["signature_present"], "signature_intact": m["signature_present"] and not m["altered"], } return {k: v for k, v in full.items() if k in visible} def infer(i): k = {} if "path_exists" in i: k["delivery_path_exists"] = i["path_exists"] if "envelope_domain" in i and POLICY[i["envelope_domain"]]: k["sender_authorized"] = i["deliverer_authorized"] if "envelope_domain" in i and "header_domain" in i: k["names_aligned"] = i["envelope_domain"] == i["header_domain"] if "envelope_recipients" in i and "header_recipients" in i: k["has_bcc_recipient"] = i["envelope_recipients"] > i["header_recipients"] if "signature_intact" in i: k["displayed_name_verified"] = i["signature_intact"] return k ENVELOPE = ("envelope_domain", "envelope_recipients", "path_exists", "deliverer_authorized") HEADER = ("header_domain", "header_recipients", "signature_present") VISIBLE = { "envelope": ENVELOPE, "envelope+header": ENVELOPE + HEADER, "envelope+header+key": ENVELOPE + HEADER + ("signature_intact",), } def measure(name): correct = wrong = unavailable = extra_round = 0 breakdown = {} for m in messages(): truth, given = oracle(m), infer(message(m, VISIBLE[name])) missing = False for decision in DECISIONS: if decision not in given: unavailable += 1 missing = True elif given[decision] == truth[decision]: correct += 1 else: wrong += 1 breakdown[decision] = breakdown.get(decision, 0) + 1 extra_round += 1 if missing else 0 return correct, wrong, unavailable, extra_round, breakdown print(f"{'visibility':<22s} {'correct':>7s} {'wrong':>6s} {'unavailable':>11s} " f"{'round':>5s} wrong breakdown") for name in VISIBLE: c, w, u, e, dk = measure(name) print(f"{name:<22s} {c:7d} {w:6d} {u:11d} {e:5d} {dk}") m = messages() misaligned = [x for x in m if x["envelope_domain"] != x["header_domain"]] print() print(f"messages {len(m)}, decisions {len(m) * len(DECISIONS)}, misaligned {len(misaligned)}, " f"all misaligned are legitimate " f"{all(x['type'] in ('proxied', 'list', 'forwarded') for x in misaligned)}") print(f"forwarded {sum(x['forwarded'] for x in m)}, " f"from a domain with a published policy {sum(POLICY[x['envelope_domain']] for x in m)}, " f"carrying a bcc recipient {sum(x['envelope_recipients'] > x['header_recipients'] for x in m)}") intact = sum(x["signature_present"] and not x["altered"] for x in misaligned) print(f"misaligned with intact signature {intact}, " f"unsigned {sum(not x['signature_present'] for x in misaligned)}, " f"altered in transit {sum(x['signature_present'] and x['altered'] for x in misaligned)}") ``` ``` visibility correct wrong unavailable round wrong breakdown envelope 60 7 133 40 {'sender_authorized': 7} envelope+header 140 7 53 40 {'sender_authorized': 7} envelope+header+key 180 7 13 13 {'sender_authorized': 7} messages 40, decisions 200, misaligned 20, all misaligned are legitimate True forwarded 14, from a domain with a published policy 27, carrying a bcc recipient 8 misaligned with intact signature 17, unsigned 2, altered in transit 1 ``` #### Reading the Numbers **The envelope governs delivery and says nothing about the header.** An intermediary that sees only the mail envelope gets 60 of the 200 decisions; it never gets the remaining 133 at all, and in forty of the forty messages it owes a round. The decisions it gets are the delivery-related ones — does a path exist, is the sender authorized. What the user will see on screen can **never** be derived from this visibility. This is the cleanest proof of the split: the information delivery needs and the information display needs are separate sets. **Reading the header buys 80 decisions and adds not a single round.** Headers are already flowing on the same connection, right after `DATA`; when the intermediary reads them, correct decisions climb from 60 to 140. This is the reverse face of the course's second claim: adding a field does not buy a decision, but **reading an already-carried field** can. What buys the gain is not the byte itself, it is being a field **whose name and meaning are fixed.** **Misalignment alone says nothing.** In 20 of 40 messages the envelope domain and the header domain differ, and in the setup **all twenty are legitimate**: proxied submission, list mail, forwarded mail. A rule that filters purely on misalignment eliminates **half** of this set. The intermediary gives the `names_aligned` decision correctly every single time — but the question it is correctly answering is not the question anyone was asking. **Path-based checking breaks with forwarding.** Seven decisions come out wrong, and all seven come from the same source: path-based checking looks at the host that **delivered** the message. In list mail the delivering host is the list's own host, even though the message genuinely came from an authorized source. The intermediary says "not authorized" and is wrong. For the 13 messages coming from a domain that does not publish a policy, the decision is **never taken at all** — the price this lesson charges for a missing declaration is not a wrong answer, it is a gap. **The signature round is paid once.** Verifying the header-covering signature costs one key-fetching round; after that round the intermediary reaches 180 correct decisions, and the number of round-indebted messages drops from 40 to **13**. The key is fetched not per message but **per domain**: since this set has a single header domain, the round is paid once and covers all forty messages. The remaining 13 come from domains that have not published a policy; their debt is closed not by the signature, but by publication. #### Why the Split Exists, Which Control Closes It The envelope and the header being separate is not a slackness added on afterward — it is a split delivery needs in order to work. Where an undeliverable message bounces back to has to be independent of the name that will be shown; otherwise a list message's bounce would fall into the original author's mailbox. A list or a forward has to be able to route the message onto another path without changing the recipient set. A blind carbon copy has to be deliverable without being written into the header. All three require the envelope to be separate from the header. The price of the split is that **nothing verifies the name written in the header.** The measurement's 20 misaligned messages are its legitimate face; the same structure means the trust the displayed name carries does not come from the protocol itself. The control that closes this has three parts, and each answers a separate question: - **Sender policy (SPF)** declares which hosts the envelope domain's mail is allowed to leave from. It is path-based, so it breaks with forwarding — all 7 of the measurement's wrong decisions come from here. If unpublished, no decision is taken: 13 messages. - **Header-covering signature (DKIM)** binds the displayed name to the domain that signs it. It survives forwarding, because it looks at content, not path; in the measurement, **17 of the 20** misaligned messages carry an intact signature. It breaks if the content is altered in transit: 1 message. Its price is one key-fetching round. - **Alignment and reporting policy (DMARC)** aligns the result of the first two **with the header domain** and declares what the domain owner wants done. It produces no proof on its own; it binds the output of two controls into a single decision. **Narrowing.** Every one of the surfaces counted closes in the same direction: **move the decision from path to content, and align the result with the header domain.** Path-based checking alone flags forwarding as false; content-based signing alone may not align with the displayed name; alignment policy alone produces nothing to verify. Together, the three raise not the **count** of decisions the intermediary takes without a round trip but their **correctness**, and in exchange they ask for one round per domain. This is exactly this lesson's measure too: the intermediary already saw the misalignment; what was missing was not seeing it, it was a declaration to bind what it saw. #### Summary - Mail is carried by two families facing opposite directions: submission pushes the message into the store (SMTP), retrieval pulls it out of the store (IMAP, POP3); the intermediary only stands on the submission road, and a decision not taken there cannot be taken back afterward. - The mail envelope governs delivery (`MAIL FROM`, `RCPT TO`), the header carries the name shown to the user (`From:`, `To:`); the two are separate fields, they do not verify each other, and a blind carbon copy exists only in the envelope. - An intermediary that sees only the envelope gets 60 of 200 decisions and owes 40 rounds; also reading the headers raises correct decisions to 140 without adding a single round. - In 20 of 40 messages the names are misaligned and all twenty are legitimate; a rule that looks only at misalignment eliminates half the set — the intermediary gives the right answer to a question no one asked. - Path-based sender policy breaks with forwarding and flags 7 decisions wrong, and produces no decision for the 13 unpublished messages; the header-covering signature binds 17 of the 20 misaligned messages and its price is a single round per domain. #### Next Step What these two lessons measured was what the intermediary could **read** from a message. Reading has one more precondition: the timestamps inside the message have to show the same clock. The next lesson measures the protocol that aligns clocks, and opens an assumption this course has not chased down for a single round yet — the assumption that the outbound delay and the return delay are equal. If they are not equal, clock offset follows directly from that gap, and the protocol has no field in its hands to see the difference. The same lesson closes the course by comparing two management models — polling devices against devices that report — on round and decision balance. ### Time and Management Protocols Source: https://academia.sh/en/courses/application-protocols/time-and-management-protocols Time synchronization's one assumption and the round's asymmetry: if the outbound and return delay are not equal, clock offset follows directly from that gap and the protocol has no field in its hands to see it; alongside this, the round and decision balance of the polling and the notifying management model. The previous two lessons measured what the intermediary could read from a message. Reading had a quiet precondition: a cache's freshness lifetime, a certificate's validity interval, a mail's `Date:` field — all of them assume the two ends' clocks are close to each other. The protocol that brings clocks close together is the purest example of the course's measurement axis: it has **four timestamps** in hand and estimates clock offset in a single round. The estimate rests on a single assumption — **that the outbound delay equals the return delay.** This lesson's question is what happens when the assumption does not hold, and whether the protocol can see that it does not. - **OP51.** The setup is M04/K03's shared setup: `station.example` and a time source. No real time server's name is written, no real network call is made. - **OP52.** The set measured is **40 rounds**. The oracle is known because we produced the setup ourselves: the server's clock is **12.500 ms** ahead of the client, and the outbound path is **4.000 ms** longer than the return path. The paths jitter round to round, but the asymmetry is systematic; the protocol does not see these two numbers, it only sees the four timestamps. - **OP53.** On the device management side the set is **40 devices** and a **20-tick** window. Every device's state changes once inside the window; the measure is whether the manager knows the correct state by the end of the window. - **OP54.** The System Administration course in M03 measured clock drift's **symptoms in logs**; that measurement is not repeated. The measure here is synchronization's **own error.** #### Four Timestamps, Two Computations Clock synchronization is done with a single exchange. Each end writes its own clock the moment it touches the message; these are the four timestamps NTP carries. ```text client server (time source) | | |-- t1: moment the client sends -------->| | t2: moment the server receives | t3: moment the server sends |<---------------------------------------| t4: moment the client receives | t1, t4 read on the client's clock t2, t3 read on the server's clock ``` This dump is not run; it is the taught form. Two computations follow from these four numbers: $$ \hat{\theta} = \frac{(t_2 - t_1) + (t_3 - t_4)}{2} \qquad \delta = (t_4 - t_1) - (t_3 - t_2) $$ The first is the **clock offset** estimate, the second is the round's total delay. The second computation is entirely independent of the clock offset: because the offset enters once as a plus and once as a minus, it cancels out. The first computation, instead, halves the sum by assuming **the outbound delay equals the return delay.** If the assumption does not hold, the error has a closed form. Let the outbound delay be $d_g$ and the return delay $d_d$; the gap between the estimate and the true offset is exactly: $$ \hat{\theta} - \theta = \frac{d_g - d_d}{2} $$ So the error does not come from noise, it comes from **the round's asymmetry**, and it equals half the gap. #### The Measurement: Asymmetry ```python SEED = 20260809 OFFSET = 0.0125 # oracle: server clock is this far ahead of the client (s) ASYMMETRY = 0.0040 # oracle: the outbound path is this much longer than the return path (s) def generator(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def rounds(count=40, seed=SEED): r, out = generator(seed), [] for i in range(count): common = 0.0150 + 0.0001 * r(9) outbound = common + ASYMMETRY / 2 + 0.00002 * r(7) inbound = common - ASYMMETRY / 2 + 0.00002 * r(7) t1 = 100.0 + i t2 = t1 + outbound + OFFSET t3 = t2 + 0.0002 t4 = t3 + inbound - OFFSET out.append((t1, t2, t3, t4)) return out def estimate(d): t1, t2, t3, t4 = d return ((t2 - t1) + (t3 - t4)) / 2, (t4 - t1) - (t3 - t2) sample = rounds() estimates = [estimate(d) for d in sample] errors = [s - OFFSET for s, _ in estimates] least_delay = min(estimates, key=lambda x: x[1]) print(f"round {len(sample)}, true offset {OFFSET * 1000:.3f} ms, " f"asymmetry {ASYMMETRY * 1000:.3f} ms") print(f"estimate average {sum(s for s, _ in estimates) / len(sample) * 1000:.4f} ms") print(f"error: average {sum(errors) / len(errors) * 1000:.4f}, " f"smallest {min(errors) * 1000:.4f}, largest {max(errors) * 1000:.4f} ms") print(f"half the asymmetry {ASYMMETRY / 2 * 1000:.4f} ms") print(f"least-delay round: estimate {least_delay[0] * 1000:.4f} ms, " f"error {(least_delay[0] - OFFSET) * 1000:.4f} ms") diff = 0.0 for t1, t2, t3, t4 in sample: offset_twin = ((t2 - t1) + (t3 - t4)) / 2 half = ((t4 - t1) - (t3 - t2)) / 2 diff = max(diff, abs(t1 + half + offset_twin - t2), abs(t3 + half - offset_twin - t4)) print(f"largest difference among the timestamps the observational twin produces: {diff:.3e} s") ``` ``` round 40, true offset 12.500 ms, asymmetry 4.000 ms estimate average 14.4917 ms error: average 1.9917, smallest 1.9400, largest 2.0400 ms half the asymmetry 2.0000 ms least-delay round: estimate 14.4700 ms, error 1.9700 ms largest difference among the timestamps the observational twin produces: 2.842e-14 s ``` #### What the Protocol Cannot See The estimates average 14.4917 ms; the true offset is 12.500 ms; the error is 1.9917 ms, and half the asymmetry is 2.0000 ms. The smallest error is 1.9400, the largest 2.0400 — all forty rounds sit **inside a narrow band around two milliseconds.** This is the proof the error is not random: if it were noise, the errors would scatter on both sides of zero. The standard improvement is measuring many rounds and picking the **least-delay** one; the round with the least queuing is considered the cleanest measurement. When that round is picked, the error drops from 1.9917 to 1.9700 ms. Filtering cuts **noise**, it does not cut **asymmetry** — because asymmetry is not noise, it is a property of the path itself. The last line is the crux. A **second world** producing the same four timestamps was constructed: in it, the path is symmetric and the server's clock really is 14.4917 ms ahead. The two worlds' timestamps agree down to $2.842 \times 10^{-14}$ seconds — which is the floating-point number's own resolution. The four timestamps **cannot tell two different truths apart**; the estimate is not miscomputed, in the twin world it is exactly correct. Because there is no field to show the asymmetry, this is not even a round debt: no matter how many rounds are spent, it does not close. Closing it lies outside the protocol — knowing from another source that the path is symmetric, or placing the timestamps before the path is entered. The course's fourth reading takes its sharpest form here: **nothing verifies what is written in the message**; here, what is written in the message is true, and it still is not enough. #### Polling Against Notifying Device management asks the same axis with a different question: how does a manager learn the state of forty devices? SNMP defines both models at once. In the **polling** model the manager asks at regular intervals, and every question is a round trip. In the **notifying** model the device reports the change itself when it happens; the unacknowledged form does not even cost a round, the acknowledged form costs one confirmation round. ```python SEED = 20260809 DEVICES, WINDOW = 40, 20 def generator(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def events(): r = generator(SEED + 1) return [{"device": i + 1, "tick": 1 + r(19), "dropped": r(9) == 0} for i in range(DEVICES)] def poll(interval): round_count = correct = delay = 0 for e in events(): round_count += WINDOW // interval seen = ((e["tick"] - 1) // interval + 1) * interval if seen <= WINDOW: correct += 1 delay += seen - e["tick"] return round_count, correct, delay / max(correct, 1) def notify(acked): dropped = sum(x["dropped"] for x in events()) return (DEVICES, DEVICES, 0.0) if acked else (0, DEVICES - dropped, 0.0) print(f"{'model':<22s} {'round':>5s} {'correct':>6s} {'wrong':>7s} {'delay':>9s}") for name, (t, d, g) in (("poll, interval 5", poll(5)), ("poll, interval 10", poll(10)), ("poll, interval 20", poll(20)), ("notify, unacked", notify(False)), ("notify, acked", notify(True))): print(f"{name:<22s} {t:5d} {d:6d} {DEVICES - d:7d} {g:6.2f} tick") print(f"\ndevices {DEVICES}, window {WINDOW} ticks, " f"notifications dropped in transit {sum(x['dropped'] for x in events())}") ``` ``` model round correct wrong delay poll, interval 5 160 40 0 2.08 tick poll, interval 10 80 40 0 4.95 tick poll, interval 20 40 40 0 10.95 tick notify, unacked 0 34 6 0.00 tick notify, acked 40 40 0 0.00 tick devices 40, window 20 ticks, notifications dropped in transit 6 ``` In the polling model, round count and detection lag trade off directly: as the interval rises from five to twenty, rounds fall from 160 to 40 and delay rises from 2.08 ticks to 10.95. The correct decision count is forty in all three — polling learns late, but it does not learn wrong. The notifying model breaks the trade-off. Its acknowledged form gives **the accuracy of the most frequent polling with 40 rounds instead of 160, and zero delay.** The unacknowledged form drives round cost to zero and produces a wrong decision on six devices. What matters about these six wrong decisions is this: the manager **cannot tell silence apart from no change**, and does not know it is wrong. The acknowledgment round buys exactly this distinction — this lesson's form of the course's rule is: **an unavailable decision is a round debt, and when the debt goes unpaid, an assumption takes the decision's place.** #### Summary - Clock synchronization derives two computations from four timestamps: the delay computation is independent of clock offset, while the offset estimate assumes the outbound and return delays are equal. - If the assumption does not hold, the error is not random: across 40 rounds the true offset is 12.500 ms while the estimates average 14.4917 ms, an error of 1.9917 ms — exactly half the asymmetry. Picking the least-delay round only brings the error down to 1.9700 ms; filtering removes noise, not asymmetry. - A symmetric twin world producing the same four timestamps can be constructed; the timestamps agree down to floating-point resolution, so the protocol has no field to see the asymmetry, and this gap does not close by spending rounds. - In management, the polling model trades rounds for delay (160 rounds and 2.08 ticks versus 40 rounds and 10.95 ticks); the acknowledged notifying model gives the same accuracy with 40 rounds and zero delay. - With unacknowledged notification the round count is zero but six devices come out wrong; the manager cannot tell silence apart from no change, and the acknowledgment round buys exactly that distinction. #### Course Wrap-Up | lesson | decision measured | taken without a round trip | wrong or unavailable | |---|---|---|---| | HTTP 01 · Methods | `repeatable`, `storable` from the method | 195/200 correct, 0 round | 5 wrong: 2 missing flag, 3 validator | | HTTP 02 · Status codes | caching and retry from the first digit | line regime 309 correct, 4004 bytes | 11 wrong; the body's 6 decisions cost 673.3 bytes/decision | | HTTP 03 · Headers and negotiation | the split between the resolved field and the carried field | 195 correct; with the full flag 13/13 correct copy | 5 wrong; 9 wrong copies without the flag; 2960 bytes zero decisions | | HTTP 04 · Cookies | the scope rule and who receives the cached response | 195 correct; scopes carry 11, 24, 40 messages | 2 wrong; with the flag wrapped, 80 decisions move to the endpoints, 40 rounds | | HTTP 05 · Versions | what framing buys the intermediary | same 195 correct, 459 bytes instead of 3861 | 189 unavailable in a context-less intermediary, 39 rounds | | HTTP 06 · Caching | the validator's resolution | tag regime 195 correct, 9 of 12 stale copies caught | the tag misses staleness in 3 copies; asking every time costs 40 rounds | | TLS 01 · Symmetric and asymmetric | the round cost of the role | 0.025 round/exchange on a single persistent connection | 1.000 round on setup for every message | | TLS 02 · Certificate chain | local verification of the chain | 33/40 correct, 0 round trip; still 0 even as the chain lengthens | 7 wrong, all from revocation; asking costs 111 rounds, same table | | TLS 03 · Handshake | the round budget and the decision the envelope charges | 40 decisions inside the envelope, none wrong | 160 unavailable, 40 rounds; bytes from 3847 to 920 | | TLS 04 · Certificate life cycle | revocation being a round debt | a client that never asks pays 0 rounds; all three options in between also have 0 round per exchange | never asking gives 22.02/40 false acceptance; asking on every exchange costs 120 rounds, 0 wrong | | OTHER 01 · WebSocket | the upgrade being a regime change | the upgrade request gives 2 of 5 decisions | 200 unavailable in the tunnel, 40 rounds, 0 wrong | | OTHER 02 · Server stream | the stream staying inside HTTP | 1 decision point, 5 decisions, 87 bytes | 195 unavailable, 39 rounds; 73 wrong across scope | | OTHER 03 · Polling | the round cost per decision | 0.200 round/decision in all four approaches | spent round plus round debt sums to 40 in all of them | | OTHER 04 · File transfer | the match between the control and data channel | single channel open 195 correct, 0 round | 8 wrong on two channels, 3 from the split; data channel wrapped costs 40 rounds | | OTHER 05 · Mail | the split between the mail envelope and the header | 60 correct with the envelope, 140 with the header, 180 with the key | 7 wrong from forwarding; unavailable drops from 133 to 13 | | OTHER 06 · Time and management | round asymmetry and the management model | acked notification 40 correct, 40 rounds, 0 delay | error 1.9917 ms, does not close by adding rounds; 6 wrong when unacked | Every number in the table is read from that row's own lesson's measurement; no row was filled in from the shared setup. The course had one sentence, and sixteen lessons measured it: **whatever is written in the message, the intermediary knows it; whatever is not written, it does not know; and whatever is written wrong, it knows wrong.** The measure stayed the same throughout — a protocol's number is not the fields it carries but how many decisions it lets the receiving side take without an extra round trip — and every unavailable decision was written down as a round debt. Three results stand at the close. **Adding a field does not buy a decision:** bytes grow, the decision table does not change; what buys the gain is not the byte itself, it is being a field whose name and meaning are fixed. **The cost of hiding is not a defect, it is a choice:** once the transport envelope closes, the intermediary loses most of the decisions, but there is no wrong decision among what remains. And **nothing verifies a decision:** the protocol trusts the declaration, because the declaration is all it has. The last lesson showed the most extreme form of this — there, the declaration was true and it still was not enough, because what was missing was not a field but a world. The intermediary that took all these decisions read the **message** throughout the course. The next course, Switching and Routing, drops down one layer and looks at the device standing there: a switch deciding which port to send a frame out of, a router deciding which neighbor to hand a packet to, and which table each of these decisions is read from. The measure shifts too — here the intermediary was counted by **what it knew**; there the device will be counted by **how correctly it forwards.** From the intermediary that takes the decision, to the device that forwards the packet.