Headers and Content Negotiation

The previous lesson measured the response’s first line and left one thing unsaid: the status line says what the response is, not which counterpart was sent. The same measurement record can be served in separate formats, separate languages, separate compressions, and all of them come back with a 200.

The place where that choice is made is the header fields. This lesson asks two questions at once. First: does adding a field to the message buy the intermediary a decision. Second: when the same address has more than one counterpart, and the field the choice was based on is not declared, which copy does the intermediary give to whom.

Two Kinds of Header Field

A header field is a name and a value separated by a colon. The dump below is for teaching purposes and is not run:

GET /olcum/kuzey HTTP/1.1
Host: station.example
Accept: application/json
Accept-Language: tr
X-Tracking-Id: trc-0004

For the intermediary, these five lines split into two sets. Host establishes its routing decision; the Accept family determines which counterpart the server produces. The last line is a field the intermediary cannot resolve: it reads the name, it reads the value, but it cannot draw a decision from it.

This split is the direct counterpart of the course’s measure. A field the intermediary cannot resolve is nothing more than bytes to it. That sentence is an intuition; this lesson’s first measurement turns it into a number.

Representation and Negotiation

An address names a resource; the resource’s counterpart as carried over the network is called a representation. The same resource can have more than one representation: the same measurement record can be delivered as a structured format, as comma-separated lines, or as plain text.

The client declares which one it wants through the Accept family of fields; the server looks at that declaration and selects a representation. This selection is called content negotiation. It has three axes, and all three use the same mechanism:

Axis	Client’s declaration	What is selected
Format	Accept	Media type
Language	Accept-Language	The text’s language
Encoding	Accept-Encoding	Compression in transit

The consequence negotiation has for the intermediary is this: a single address no longer corresponds to a single response. When the intermediary stores a copy, it also has to know which request that copy was produced for. If it does not know, it hands the stored copy to the wrong client.

The Requester’s Declaration, the Server’s Declaration

Negotiation runs on two declarations, and they point in separate directions. The request-side Accept family lists what the client can accept; it is a preference list, not a single choice. The response-side Content-Type declares what the server actually sent, and it is singular.

The intermediary needs both, for separate reasons. The request-side declaration lets it test whether the copy it holds matches this request. The response-side declaration lets it know what the copy it stored actually is. Neither works without the other: knowing what the client wants does not build a match without knowing what is on hand.

The two also fail differently. Because Accept is a preference list, the server may be unable to produce any option on it; negotiation then comes back empty, and the decision returns to the client. Because Content-Type is a single value, it can be written wrong: when the server sends one format but declares another, the intermediary stores it under the wrong declaration and binds the copy to the wrong key.

Encoding parts ways with the other two axes at one point. Format and language touch the representation’s meaning: a different format is a different record layout, a different language is a different text. Encoding leaves the meaning untouched and changes only the transport-level representation — a compressed record, decompressed, is identical to its uncompressed counterpart. The intermediary still cannot ignore it: handing a compressed copy to a client that never declared it could accept compression produces a response that client cannot read. Meaning stays the same; the key still has to be kept separate.

Assumptions of the two measurements:

• HA13 — The same forty exchanges and five decisions are used; a separate generator assigns the format and language each exchange requests. oracle and infer are untouched.
• HA14 — The three added fields carry real information but never enter a branch in the intermediary’s code; what is measured is not a field’s size but whether the reader can branch on it.
• HA15 — The second measurement runs only over storable exchanges: safe method, not personalized. The rest of the forty never enter the intermediary’s store.
• HA16 — The intermediary stores one copy per key, and only the declared fields join the address in that key; the first copy to arrive holds its place and is given out to every later request.
• HA17 — The representation the client wants is known because we generated it ourselves; the copy handed out is counted right or wrong against this known request.
• HA18 — The declared vary is tried at three values: none, format only, format and language. No mechanism tests the declaration’s accuracy; the intermediary believes whatever fields the server lists.

First Measurement: Adding a Field

The first measurement takes the five-decision set built by the previous two lessons as is and changes exactly one thing: three fields the intermediary cannot resolve are added to the message. The oracle and infer functions are untouched.

SEED = 20260809
SAFE = {"GET", "HEAD"}
IDEMPOTENT = {"GET", "HEAD", "PUT", "DELETE"}
METHOD_POOL = ["GET", "GET", "GET", "HEAD", "POST", "PUT", "DELETE"]
PATH_POOL = ["/olcum/kuzey", "/olcum/yamac", "/ozet", "/kayit", "/oturum"]
FORMAT_POOL = ["json", "json", "csv", "text", "json"]
LANGUAGE_POOL = ["tr", "tr", "en", "tr", "en"]
DECISIONS = ("storable", "shareable", "fresh", "repeatable", "redirectable")
VISIBLE = {"open": ("method", "path", "host", "private_marker", "validator")}
VISIBLE["open+extra"] = VISIBLE["open"] + ("tracking_id", "client_version", "region")


def generator(seed: int):
    d = seed % 2147483646 + 1

    def r(n: int) -> int:
        nonlocal d
        d = (d * 48271) % 2147483647
        return d % n
    return r


def exchanges(count: int = 40, seed: int = SEED) -> list[dict]:
    r, out = generator(seed), []
    for i in range(count):
        method, path = METHOD_POOL[r(7)], PATH_POOL[r(5)]
        private = path == "/oturum" or r(5) == 0
        stale = r(3) == 0
        out.append({
            "num": i + 1, "method": method, "path": path,
            "private": private, "stale": stale,
            "private_marker": private and r(4) != 0,
            "tag_missed": stale and r(6) == 0,
        })
    return out


def assign_preferences(items: list[dict], seed: int = SEED + 211) -> list[dict]:
    r = generator(seed)
    for a in items:
        a["format"], a["language"] = FORMAT_POOL[r(5)], LANGUAGE_POOL[r(5)]
    return items


def oracle(a: dict) -> dict:
    return {
        "storable": a["method"] in SAFE and not a["private"],
        "shareable": not a["private"],
        "fresh": not a["stale"],
        "repeatable": a["method"] in IDEMPOTENT,
        "redirectable": True,
    }


def message(a: dict, visible: tuple) -> dict:
    full = {"method": a["method"], "path": a["path"], "host": "station.example",
            "private_marker": a["private_marker"],
            "validator": a["stale"] and not a["tag_missed"],
            "tracking_id": f"trc-{a['num']:04d}", "client_version": "measure-client",
            "region": "north-slope"}
    return {k: v for k, v in full.items() if k in visible}


def byte_count(i: dict) -> int:
    return sum(len(k) + len(str(v)) + 4 for k, v in i.items())


def infer(i: dict) -> dict:
    k = {}
    if "method" in i:
        k["repeatable"] = i["method"] in IDEMPOTENT
    if "private_marker" in i:
        k["shareable"] = not i["private_marker"]
        if "method" in i:
            k["storable"] = i["method"] in SAFE and not i["private_marker"]
    if "validator" in i:
        k["fresh"] = not i["validator"]
    if "host" in i:
        k["redirectable"] = True
    return k


def measure(label: str, batch: list[dict]) -> tuple:
    d = y = e = t = b = 0
    for a in batch:
        truth, given = oracle(a), infer(message(a, VISIBLE[label]))
        missing = False
        for decision in DECISIONS:
            if decision not in given:
                e += 1
                missing = True
            elif given[decision] == truth[decision]:
                d += 1
            else:
                y += 1
        t += 1 if missing else 0
        b += byte_count(message(a, VISIBLE[label]))
    return d, y, e, t, b


def negotiate(batch: list[dict], vary: tuple) -> tuple:
    """An intermediary that stores one copy per address; only the declared fields enter the key."""
    store, correct, wrong, vary_bytes = {}, 0, 0, 0
    for a in batch:
        if not (a["method"] in SAFE and not a["private"]):
            continue
        vary_bytes += byte_count({"vary": vary}) if vary else 0
        key = (a["path"],) + tuple(a[field] for field in vary)
        requested = (a["format"], a["language"])
        served = store.setdefault(key, requested)
        if served == requested:
            correct += 1
        else:
            wrong += 1
    return correct, wrong, len(store), vary_bytes

The three added fields are a tracking ID, a client version, and a region name. All three carry real information, and none of the three enters a branch in the infer function.

batch = assign_preferences(exchanges())
print(f"{'visibility':<10s} {'correct':>7s} {'wrong':>7s} {'unavailable':>11s} "
      f"{'extra round':>13s} {'bytes':>6s}")
for label in ("open", "open+extra"):
    d, y, e, t, b = measure(label, batch)
    print(f"{label:<10s} {d:7d} {y:7d} {e:11d} {t:13d} {b:6d}")
ba, be = measure("open", batch)[4], measure("open+extra", batch)[4]
print(f"fields {len(VISIBLE['open'])} -> {len(VISIBLE['open+extra'])}, "
      f"bytes {ba} -> {be}, increase {100 * (be - ba) / ba:.0f}%, decisions gained "
      f"{measure('open+extra', batch)[0] - measure('open', batch)[0]}")

visibility correct   wrong unavailable   extra round  bytes
open           195       5           0             0   3724
open+extra     195       5           0             0   6764
fields 5 -> 8, bytes 3724 -> 6764, increase 82%, decisions gained 0

All four columns of the decision table stay the same: 195 correct, 5 wrong, 0 unavailable, 0 extra round trips. Bytes, though, rise from 3724 to 6764 — an 82 percent increase, zero decisions. Three thousand and forty extra bytes were carried across the forty exchanges, and nothing was gained in return.

This is the course’s second claim, and its price is paid here: adding a field does not buy a decision. A field’s value comes not from the information it carries but from whether the reader can build a branch on that information. A tracking ID may be valuable to the measurement station; to the intermediary it is three hundred sixty bytes, and nothing more.

Second Measurement: The Cost of Negotiation

The second measurement runs in the opposite direction. Now the added field is one the intermediary can resolve, and the question is: how many decisions does declaring which field entered negotiation buy.

The intermediary binds the copy it stores to a key. Without a declaration, the key is just the address: the first copy to arrive is stored and given out to every later request. With a declaration, the declared fields also enter the key, and every representation is stored separately. The measurement runs only over storable exchanges — safe method, not personalized.

This declaration’s protocol counterpart is the Vary header field: the server lists which header fields it looked at while producing the response. The model’s vary tuple is exactly this list. The field sits on the response side for a reason — only the party making the selection knows what entered negotiation, and the intermediary knows it only if told.

The list’s length has a limit, visible in the measurement’s last column. Every field that enters the key multiplies the number of stored copies. If a field whose value differs for every client enters the list, every request produces a separate key, and the intermediary never reuses a single copy. The correct declaration lists exactly the fields that entered negotiation and nothing else: too little produces a wrong copy, too much produces a copy that is never reused.

stored = [a for a in batch if a["method"] in SAFE and not a["private"]]
print(f"exchanges entering negotiation {len(stored)}, "
      f"requested representations {len({(a['format'], a['language']) for a in stored})} kinds, "
      f"addresses {len({a['path'] for a in stored})}")
print()
print(f"{'declared vary':<22s} {'correct copy':>12s} {'wrong copy':>11s} "
      f"{'stored':>7s} {'vary bytes':>10s}")
for vary in ((), ("format",), ("format", "language")):
    d, y, n, vb = negotiate(batch, vary)
    print(f"{str(vary):<22s} {d:12d} {y:11d} {n:7d} {vb:10d}")

d0, y0, n0, b0 = negotiate(batch, ())
d2, y2, n2, b2 = negotiate(batch, ("format", "language"))
print()
print(f"full vary: decisions gained {d2 - d0}, bytes spent {b2 - b0}, "
      f"{(b2 - b0) / (d2 - d0):.1f} bytes per decision; stored copies {n0} -> {n2}")

exchanges entering negotiation 13, requested representations 6 kinds, addresses 4

declared vary          correct copy  wrong copy  stored vary bytes
()                                4           9       4          0
('format',)                       9           4       8        247
('format', 'language')           13           0      11        390

full vary: decisions gained 9, bytes spent 390, 43.3 bytes per decision; stored copies 4 -> 11

There are thirteen storable exchanges, four addresses, and six kinds of requested representation. The measurement band’s lower bound in this set is $1/13 = 0.077$; no difference smaller than that is claimed.

With no declaration at all, the intermediary stores four copies and hands out the wrong copy to nine of the thirteen requests. A wrong copy is not a broken response: it is a valid, fresh representation from the correct address — it just does not belong to the requester. A requester expecting plain text gets a structured record; one expecting Turkish gets English.

When only format is declared, wrong copies drop from nine to four. The remaining four come from the language axis: the intermediary has learned to separate by format, not by language. An incomplete declaration is a smaller version of no declaration at all, because missing even a single axis is enough to make a copy wrong. When both axes are declared, wrong copies drop to zero.

Reading the Two Measurements Side by Side

Both measurements add a field to the same message, and their outcomes come out diametrically opposed.

Added	Bytes	Decisions gained
Three unresolvable fields	3040	0
Declared vary	390	9

The difference is not in the fields’ size but in what the reader can do with them. The three fields in the first measurement add no branch to the intermediary’s code at all; the single field in the second changes its key. This is why the measure has to be decisions, not field count: field count would have counted both as “something was added to the message.”

Negotiation has a second cost, and it is not measured in bytes. Once the declaration is complete, the number of copies the intermediary stores rises from four to eleven. The same address is no longer a single entry for the intermediary; each representation takes its own slot and is refreshed on its own. The price of handing out the right copy is storing the same resource more than once.

The fourth reading holds here too. The intermediary cannot test which fields actually entered negotiation. If the server under-declares, it stores under an incomplete key and distributes the wrong copy without raising a single error. No mechanism checks the declaration’s accuracy; the intermediary believes it. Every gain and cost counted in this lesson rests on that single assumption.

Summary

• Header fields split into two groups as far as the intermediary is concerned: ones that resolve into a decision, and ones that are only carried.
• On the set of forty exchanges, adding three unresolvable fields leaves the decision table unchanged — 195 correct, 5 wrong — but bytes rise from 3724 to 6764, an 82 percent increase.
• An address names a resource; content negotiation is the same resource having more than one representation across the format, language, and encoding axes.
• If the field entering negotiation is not declared, the intermediary hands out the wrong copy in nine of thirteen storable exchanges; declaring format alone leaves four, declaring both axes leaves zero.
• Added to the same message, 3040 bytes buy zero decisions and 390 bytes buy nine; this is why the measure is decisions, not field count.
• The price of handing out the right copy is not limited to bytes: the number of stored copies rises from four to eleven.

Next Step

Content negotiation means the same address has more than one representation, and the intermediary derives which copy goes to whom from the header fields. Every representation measured in this lesson shared one property: two clients making the same declaration deserved the same copy. Format and language describe not who the requester is, but what they want.

So what happens when the response is personalized — when two clients arriving at the same address with the same format and language declaration deserve different responses? Then the field needed to select the right copy has to carry not what was requested but who it belongs to. The next lesson takes up that field and the intermediary’s relationship with it.