Closed- and Open-Weight Models

Every scaffolding added in the previous lesson was built around the model, and access to the model was assumed to be given. That access is itself a decision. A language model can be reached only through a remote model endpoint; or the model’s weights are held and run on your own hardware. The first is called closed-weight access and the second open-weight access.

This distinction is usually discussed as a debate over preference, and the debate most often ends where both sides are right, because the sides speak from different criteria. This lesson turns it into numbers on three criteria: the number of settings that stay in hand, the token cost of finishing the same job, and the data that crosses the organization boundary. All three are quantities an engineer can count from their own side, without ever looking inside the model. No model, provider or license name appears, because what is measured is not a product but the structure of the access mode; names change in five years, the structure does not.

• LM58. The task set has the same structure and the simulator is the same as before: 24 fictional support requests, now with English texts, are divided into five classes, and the measure is the hit rate. The requests, the classes and the readings are fictional.
• LM59. The two access modes are modeled with a single difference: the width of the simulator’s familiar piece set. It is wide in the closed form and narrow in the open form. This is the assumption of the fiction and not a claim about the capability of real models; what is measured is how the difference turns into cost.
• LM60. The scaffolding of the open form is adding examples. Each fictional example placed in the prompt costs 23 tokens and widens the familiar piece set by one step.
• LM61. The control held is measured in a fictional list of nine settings, by counting in which form each setting can be decided.
• LM62. The data boundary is measured with three numbers: the number of requests that cross the organization boundary, the number of tokens and the number of numeric fields. The numeric fields are fictional subscriber numbers and meter readings; this criterion does not count the nature of the data, it counts the amount that crosses the boundary.
• LM63. Uncertainty is measured with two runs; the second run is taken with seed 20260219. Because the two forms share the same simulator, they are expected to give the same number at the same width, and this is used as a checkpoint in the table.
• LM64. The criteria are read separately and not added up. The number of settings, the number of tokens and the number of requests crossing the boundary are not converted into a common unit; the winner of each criterion is written separately.

Unit Cost: What the Same Hit Rate Costs

# SHARED -- FICTIONAL. Both access modes are built on the same simulator; no model, provider or
# license is named. The only difference in the fiction is the width of the familiar piece set.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def tokenize(text):
    """Fictional tokenizer: letter pieces are split into fours, digit pieces into twos."""
    out, buffer = [], ""
    for ch in text + " ":
        if ch.isalnum():
            buffer += ch
            continue
        if buffer:
            n = 2 if buffer.isdigit() else 4
            out += [buffer[i:i + n] for i in range(0, len(buffer), n)]
            buffer = ""
        if ch.strip():
            out.append(ch)
    return out


MAIN = [("meter not read estimated invoice issued", "reading"),
        ("invoice amount doubled compared to last month", "payment"),
        ("water pressure dropped during the night", "water"),
        ("planned outage notice did not arrive", "outage"),
        ("subscription transfer was not applied to meter", "subscription")]
EXTRA = ["meter number may have been changed", "address information looks old",
         "same problem did not occur last month", "cannot reach the meter door closed",
         "outage lasted longer than expected", "no automatic payment instruction on meter"]
LEXICON = {"outage": ["outa", "noti"], "reading": ["read", "meas", "esti"],
           "water": ["pres", "drop", "tank", "pipe"],
           "payment": ["invo", "bill", "paym", "amou"],
           "subscription": ["subs", "addr", "tran", "cont"]}
CLASSES = list(LEXICON)
EXAMPLE_TOKENS = 23                             # cost of each fictional example added to the prompt
REQUESTS = []
for i in range(24):
    r = rng(SEED + 31 * i)
    a, b = 100000 + int(r() * 380000), 100000 + int(r() * 380000)
    REQUESTS.append({"id": 4100 + i, "label": MAIN[i % 5][1],
                     "text": f"subscriber {4100 + i}: {MAIN[i % 5][0]}, {EXTRA[(5 * i + 2) % 6]}."
                             f" main meter {a} sub meter {b}."})


def call(t, width, seed):
    """Simulator: picks a label from the pieces it knows; returns ambiguous if the top two tie."""
    tokens, known = tokenize(t["text"]), {s: set(o[:width]) for s, o in LEXICON.items()}
    p = sorted(((sum(b in known[s] for b in tokens), s) for s in CLASSES), key=lambda x: (-x[0], x[1]))
    if p[0][0] == 0 or p[0][0] == p[1][0]:
        return "ambiguous"                      # both labels are written at once, the format check fails
    r = rng(seed + 1009 * width + 7 * t["id"])
    return p[1][1] if r() < 0.14 else p[0][1]


def measure(width, examples, seed=SEED):
    correct = token_count = 0
    for t in REQUESTS:
        correct += call(t, width, seed) == t["label"]
        token_count += len(tokenize(t["text"])) + examples * EXAMPLE_TOKENS
    return correct / 24, token_count / 24


print("access mode     examples  hit rate A  hit rate B  calls   tokens/request")
for name, w, e in (("closed-weight", 4, 0), ("open-weight", 1, 0),
                   ("open-weight", 2, 2), ("open-weight", 3, 4),
                   ("open-weight", 4, 6)):
    a, tokens = measure(w, e)
    b = measure(w, e, SEED + 1)[0]
    print(f"{name:<16}{e:<10}{a:<12.4f}{b:<12.4f}{'1.0000':<8}{tokens:.1f}")

access mode     examples  hit rate A  hit rate B  calls   tokens/request
closed-weight   0         0.7917      0.7083      1.0000  41.8
open-weight     0         0.1250      0.1667      1.0000  41.8
open-weight     2         0.3750      0.3750      1.0000  87.8
open-weight     4         0.5833      0.6667      1.0000  133.8
open-weight     6         0.7917      0.7083      1.0000  179.8

In the bare call the two forms spend the same number of tokens and do not give the same number: closed 0.7917, open 0.1250. The difference comes not from the model itself but from the narrow piece set of the simulator that runs in the open form; this is the assumption of the fiction. What matters is what follows: the difference is one that can be closed, and the price of closing it is written in tokens.

As two examples are added to the prompt at a time, the hit rate climbs the steps 0.3750, 0.5833 and 0.7917. At the sixth example the open form catches up exactly with the number of the closed form: 0.7917 against 0.7917, and in the second run 0.7083 against 0.7083. The two forms now do the same job. The only thing they do not do is do it at the same price: 41.8 tokens per request against 179.8 tokens. The number of calls is one in both, and the latency class is a single call in both; the difference lies entirely in the length of the prompt.

The place to read in the ladder is this: the cost moved from the model to the prompt. In the open-weight form, what the model does not know is retold in every request. This is the appearance in this lesson of the general law of scaffolding: what is missing in one place is paid for in tokens in another place, and that payment is repeated in every request. In a set of twenty-four requests the difference is 3,312 tokens; as the number of requests grows the difference grows in direct proportion, because unless the examples are cached they are sent again in every call.

There is also a number that does not appear in the table. When the scores of the top two candidates are equal, the simulator writes both labels at once instead of a single label and fails the format check; these responses are counted as misses. With the narrow piece set this happens often, with the wide set it is rare. So part of the low number of the open form comes not from a wrong label but from unusable output. When the measure is reduced to a single number this distinction disappears, yet the two kinds of failure mean different things on the product side: a wrong label silently starts a wrong job, while a malformed output is a request that stops.

Control Held and the Data Boundary

SETTINGS = [("the weights themselves", 0, 1), ("tokenizer", 0, 1), ("version pinning", 0, 1),
            ("context window limit", 0, 1), ("rate limit", 0, 1), ("rollback during an outage", 0, 1),
            ("distribution temperature", 1, 1), ("stop condition", 1, 1), ("output length limit", 1, 1)]
CLOSED, OPEN = sum(c for _, c, _ in SETTINGS), sum(o for _, _, o in SETTINGS)
print(f"{'setting':<26}{'closed-weight':<15}open-weight")
for name, c, o in SETTINGS:
    print(f"{name:<26}{'held' if c else '-':<15}{'held' if o else '-'}")
print(f"criterion 1 -- settings held: closed {CLOSED}/{len(SETTINGS)}, open {OPEN}/{len(SETTINGS)}")
ids = sum(sum(w.isdigit() for w in t["text"].replace(":", " ").replace(".", " ").split())
          for t in REQUESTS)
outside = sum(len(tokenize(t["text"])) for t in REQUESTS)
print(f"criterion 3 -- beyond the boundary: closed {len(REQUESTS)}/{len(REQUESTS)} requests,"
      f" {outside} tokens, {ids} numeric fields")
print(f"{'':36}open   0/{len(REQUESTS)} requests, 0 tokens, 0 numeric fields")

setting                   closed-weight  open-weight
the weights themselves    -              held
tokenizer                 -              held
version pinning           -              held
context window limit      -              held
rate limit                -              held
rollback during an outage -              held
distribution temperature  held           held
stop condition            held           held
output length limit       held           held
criterion 1 -- settings held: closed 3/9, open 9/9
criterion 3 -- beyond the boundary: closed 24/24 requests, 1004 tokens, 72 numeric fields
                                    open   0/24 requests, 0 tokens, 0 numeric fields

Three of the nine settings stay in hand under closed-weight access: the distribution temperature, the stop condition and the output length limit. These are the settings sent along with the request. The remaining six are not in hand, and they all have something in common: the weights, the tokenizer, the version, the window limit, the rate limit and returning to the old behavior at the moment of an outage are decisions that belong to the place where the model runs. If the model runs on someone else’s machine, those decisions are with someone else. Under open-weight access all nine stay in hand, because the party that runs the model and the party that decides are the same.

An invisible consequence of this is that the version cannot be pinned. The fabrication rate measured in the seventh lesson, the task hit rate measured in the sixth lesson and the 0.7917 measured in this lesson all belong to a particular behavior. When that behavior changes without notice, all the measures must be taken again and none of the old numbers is valid. Under closed-weight access the lifetime of a measurement is bounded by the lifetime of a decision that is not in hand.

The third criterion is read in the same table. In the closed form, 24 of the twenty-four requests, 1004 tokens and 72 numeric fields cross the organization boundary; these numeric fields are fictional subscriber numbers and meter readings. In the open form all three numbers are zero. This criterion is not a difference of degree between the two but a difference of threshold: the data that crosses the boundary either exists or it does not.

The table should not be read one way only. In return for the six settings that are not in hand under closed-weight access, there is a load that does not pass to the organization: running the model, setting aside hardware, moving the version and absorbing the outage. Under open-weight access these nine settings stay in hand because all nine responsibilities stay in hand. When the criterion is named “control held” this symmetry becomes invisible; next to the number, what it brings must be written as much as what it burdens. What this lesson measures is not which form is good, but the criteria on which the two forms part.

Where the Choice Changes

cb, ob = measure(4, 0), measure(4, 6)
print("criterion      closed-weight       open-weight         winner")
print(f"control        {CLOSED}/9 settings        {OPEN}/9 settings        open")
print(f"unit cost      {cb[1]:.1f} tokens         {ob[1]:.1f} tokens        closed")
print(f"data boundary  24/24 requests      0/24 requests       open")
print(f"for the same hit rate {cb[0]:.4f} the cost ratio is {ob[1] / cb[1]:.2f}")
print(f"two criteria pick the open-weight form, one picks the closed-weight form;"
      f" no common unit")

criterion      closed-weight       open-weight         winner
control        3/9 settings        9/9 settings        open
unit cost      41.8 tokens         179.8 tokens        closed
data boundary  24/24 requests      0/24 requests       open
for the same hit rate 0.7917 the cost ratio is 4.30
two criteria pick the open-weight form, one picks the closed-weight form; no common unit

The choice changes on the unit cost criterion. While control and the data boundary pick the open-weight form, unit cost picks the closed form, and the difference is not small: 4.30 times the tokens for the same hit rate. Because two criteria are on one side and one on the other, saying “the majority picks open” would be wrong, since the three numbers have no common unit. The number of settings and the number of tokens cannot be added; the number of requests crossing the boundary cannot be compared with either of the two.

This is not a deficiency but the correct reading of the measurement. The decision is made by weighting the criteria, and what gives the weight is the context of the system. In an organization that cares less about the latency of the response than about where the data goes, the third criterion alone decides and the 4.30-fold cost is accepted. In an organization with a tight token budget the second criterion dominates. The same three numbers support two different decisions as the weights change; what changes is not the measurement but the weight.

The rule of the course works here for the second time: a given component is measured from its surface. None of the numbers measured in this lesson came from inside the model. The number of settings was read from what the interface allows, the number of tokens from what was put in the prompt, the data that crosses the boundary from where the request goes. All three are traces left outside by a component whose inside cannot be seen, and all three are things an engineer can count from their own code. This is the gain of writing these three numbers instead of calling an access mode “stronger” or “safer”: descriptions are argued over, numbers are compared. And when the cost is not written next to the number, the comparison turns back into an argument — 0.7917 looks the same in both rows, and the 4.30-fold difference between them exists only in the cost column.

Summary

• Closed-weight access leaves 3 of the nine settings in hand and open-weight access leaves 9; all six settings that are not in hand belong to the place where the model runs.
• In the bare call the hit rate is 0.7917 against 0.1250; when six examples are added to the prompt the open form catches up with the number of the closed form in both runs (0.7917 and 0.7083).
• The cost of the same hit rate is 41.8 tokens per request against 179.8 tokens, a ratio of 4.30 times; the number of calls and the latency class are the same in both.
• On the data boundary, in the closed form 24/24 requests, 1004 tokens and 72 numeric fields cross the organization boundary and in the open form none does; this is a difference of threshold, not of degree.
• The choice changes on the unit cost criterion, and because the three numbers have no common unit, the decision is made by weighting.

Next Step

This lesson compared two forms and stopped where the decision is left to the weights. In a real choice there are not two options and there are not three criteria: within the same access mode there are candidates that differ from one another on the axes of capability, latency and token cost, and privacy stands above them as a separate criterion. The next lesson builds three candidates at the simulator level, counts them on four criteria and answers a single question: when the weights change, does the winner change, and if it does, at which threshold? The weight sweep and the normalization procedure were measured in the Environment Selection lesson of the Data Analysis with Python course; they will not be repeated here, only their result will be used.