Model Selection

The previous lesson compared two access modes on three criteria and stopped where the decision is left to the weights. In a real choice there are not two options. There are several models that could do the same job; they differ from one another on the axes of capability, token cost, latency and where the data goes, and none of them is best on all four axes at once.

This lesson builds three candidates at the simulator level, counts them on four criteria and answers a single question: when the weights change, does the winner change, and if it does, at which threshold? How a weight sweep and the choice of a normalization rule can turn the result was measured in the Environment Selection lesson of the Data Analysis with Python course; that procedure is not repeated here, a single normalization rule is used and its result is read.

• LM65. The three candidates are three settings of a single simulator. They differ in the width of the familiar piece set, the unit multiplier and the length of the response they produce. No model, version or provider is named; the candidates are called small, medium and large capability and are fictional.
• LM66. The cost unit is the equivalent token. A token of a candidate is scaled by a fictional multiplier that says how many times the account of the smallest candidate’s token it holds: small 1, medium 4, large 12. No currency is written.
• LM67. Latency is measured in the number of steps: the average number of calls per request times the number of output tokens produced. Raw time is not written, because generation is sequential and the number of steps does not change from run to run.
• LM68. The small and medium candidates run on their own hardware with open-weight access, and the large candidate is called only through a model endpoint; the number of requests crossing the organization boundary is 0, 0 and 24 respectively.
• LM69. The small candidate has a scaffolding: for responses that fail the format check, two fictional examples are added to the prompt and the request is asked again. This scaffolding raises the hit rate and enlarges the number of calls and tokens.
• LM70. The normalization rule is single: in each criterion the worst candidate gets 0 and the best candidate gets 1, and the values in between are scaled to the range. The weights are applied to these normalized numbers.

Three Candidates and Four Criteria

# SHARED -- FICTIONAL. The three candidates are built on the same simulator; no model or
# provider is named. The candidates differ in familiar piece width, unit multiplier and output
# length. The cost unit is EQUIVALENT TOKENS: number of tokens times the candidate's unit multiplier.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def tokenize(text):
    """Fictional tokenizer: letter pieces are split into fours, digit pieces into twos."""
    out, buffer = [], ""
    for ch in text + " ":
        if ch.isalnum():
            buffer += ch
            continue
        if buffer:
            n = 2 if buffer.isdigit() else 4
            out += [buffer[i:i + n] for i in range(0, len(buffer), n)]
            buffer = ""
        if ch.strip():
            out.append(ch)
    return out


MAIN = [("meter not read estimated invoice issued", "reading"),
        ("invoice amount doubled compared to last month", "payment"),
        ("water pressure dropped during the night", "water"),
        ("planned outage notice did not arrive", "outage"),
        ("subscription transfer was not applied to meter", "subscription")]
EXTRA = ["meter number may have been changed", "address information looks old",
         "same problem did not occur last month", "cannot reach the meter door closed",
         "outage lasted longer than expected", "no automatic payment instruction on meter"]
LEXICON = {"outage": ["outa", "noti"], "reading": ["read", "meas", "esti"],
           "water": ["pres", "drop", "tank", "pipe"],
           "payment": ["invo", "bill", "paym", "amou"],
           "subscription": ["subs", "addr", "tran", "cont"]}
CLASSES = list(LEXICON)
REQUESTS = []
for i in range(24):
    r = rng(SEED + 31 * i)
    a, b = 100000 + int(r() * 380000), 100000 + int(r() * 380000)
    REQUESTS.append({"id": 4100 + i, "label": MAIN[i % 5][1],
                     "text": f"subscriber {4100 + i}: {MAIN[i % 5][0]}, {EXTRA[(5 * i + 2) % 6]}."
                             f" main meter {a} sub meter {b}."})
# (name, piece width, unit multiplier, output tokens, example cost, requests crossing the boundary)
CANDIDATES = [("small", 2, 1, 5, 46, 0), ("medium", 3, 4, 7, 0, 0), ("large", 4, 12, 14, 0, 24)]


def call(t, width, seed):
    """Simulator: picks a label from the pieces it knows; returns ambiguous if the top two tie."""
    tokens, known = tokenize(t["text"]), {s: set(o[:width]) for s, o in LEXICON.items()}
    p = sorted(((sum(b in known[s] for b in tokens), s) for s in CLASSES), key=lambda x: (-x[0], x[1]))
    if p[0][0] == 0 or p[0][0] == p[1][0]:
        return "ambiguous"
    r = rng(seed + 1009 * width + 7 * t["id"])
    return p[1][1] if r() < 0.14 else p[0][1]


def measure(candidate, seed=SEED):
    name, width, multiplier, output, example, outside = candidate
    correct = calls = inputs = 0
    for t in REQUESTS:
        n = len(tokenize(t["text"]))
        y, calls, inputs = call(t, width, seed), calls + 1, inputs + n
        if y == "ambiguous" and example:        # scaffolding: add two examples to the prompt and ask again
            y, calls, inputs = call(t, width + 1, seed), calls + 1, inputs + n + example
        correct += y == t["label"]
    calls, inputs = calls / 24, inputs / 24
    return {"name": name, "hit_rate": correct / 24, "equivalent": (inputs + calls * output) * multiplier,
            "steps": calls * output, "calls": calls, "outside": outside}


S = [measure(c) for c in CANDIDATES]
print("candidate  hit rate  hit rate B  equivalent tokens  steps  calls/request  crossing")
for c, s in zip(CANDIDATES, S):
    b = measure(c, SEED + 1)["hit_rate"]
    print(f"{s['name']:<11}{s['hit_rate']:<10.4f}{b:<12.4f}{s['equivalent']:<19.1f}"
          f"{s['steps']:<7.2f}{s['calls']:<15.4f}{s['outside']}/24")

candidate  hit rate  hit rate B  equivalent tokens  steps  calls/request  crossing
small      0.5000    0.5833      89.5               7.29   1.4583         0/24
medium     0.5833    0.6667      195.3              7.00   1.0000         0/24
large      0.7917    0.7083      670.0              14.00  1.0000         24/24

The four columns show four different winners. On hit rate the large candidate leads: 0.7917. On equivalent tokens the small candidate leads: 89.5, that is, less than a seventh of the large one’s 670.0. On steps the medium candidate leads, but the margin is thin: 7.00 against 7.29. On requests crossing the organization boundary the small and medium candidates lead together, because both run on their own hardware.

Where the small candidate loses on the number of steps is written in the calls column: 1.4583 calls per request. This is the cost of the scaffolding. For responses that fail the format check, two examples are added to the prompt and the request is asked again; the hit rate rises to 0.5000 this way, but every request that is asked again adds one more step. The scaffolding takes the hit rate and gives up latency, and both numbers stand side by side in the table.

Attention must be paid to what the equivalent token unit is. The three candidates receive the same request and read almost the same number of tokens; by raw token count all three are close to one another. The difference lies not in the number of the token but in what it holds: processing the same token means twelve times the computational load on the large candidate compared with the small one. The unit multiplier reduces this load to a single number and makes the comparison possible. This is also the reason no currency is written: the multiplier is structural and reads the same way five years from now, whereas a price list does not. The output tokens are counted separately, because the output is produced sequentially and enters both the cost and the steps; a 14-token response from the large candidate means twice the latency of the small candidate, which gives the same response in 5 tokens.

When the Weight Changes, the Winner Changes

CRITERIA = [("hit rate", "hit_rate", 1), ("equivalent tokens", "equivalent", -1), ("steps", "steps", -1),
            ("requests crossing", "outside", -1)]


def normalize(results, key, direction):
    """Single normalization rule: in the range the worst is 0, the best is 1."""
    v = [s[key] for s in results]
    lo, hi = min(v), max(v)
    if hi == lo:
        return [1.0] * len(v)
    return [((x - lo) if direction > 0 else (hi - x)) / (hi - lo) for x in v]


N = {k: normalize(S, k, d) for _, k, d in CRITERIA}
WEIGHTS = [("balanced", (1, 1, 1, 1)), ("hit-rate weighted", (4, 1, 1, 1)),
           ("cost weighted", (1, 4, 1, 1)), ("latency weighted", (1, 1, 4, 1)),
           ("data-boundary weighted", (1, 1, 1, 4))]
print(f"{'criterion':<20}{'small':<8}{'medium':<8}large")
for name, k, _ in CRITERIA:
    print(f"{name:<20}{N[k][0]:<8.4f}{N[k][1]:<8.4f}{N[k][2]:.4f}")
print(f"{'weight set':<24}{'small':<8}{'medium':<8}{'large':<8}winner")
for name, w in WEIGHTS:
    p = [sum(w[j] * N[k][i] for j, (_, k, _) in enumerate(CRITERIA)) for i in range(3)]
    print(f"{name:<24}{p[0]:<8.4f}{p[1]:<8.4f}{p[2]:<8.4f}{S[p.index(max(p))]['name']}")

criterion           small   medium  large
hit rate            0.0000  0.2857  1.0000
equivalent tokens   1.0000  0.8176  0.0000
steps               0.9583  1.0000  0.0000
requests crossing   1.0000  1.0000  0.0000
weight set              small   medium  large   winner
balanced                2.9583  3.1033  1.0000  medium
hit-rate weighted       2.9583  3.9605  4.0000  large
cost weighted           5.9583  5.5562  1.0000  small
latency weighted        5.8333  6.1033  1.0000  medium
data-boundary weighted  5.9583  6.1033  1.0000  medium

Across five weight sets, three different winners come out. With equal weights the medium candidate wins, and the margin is small: 3.1033 against 2.9583. When the hit-rate weight is raised to four, the large candidate moves ahead, but again by a hair: 4.0000 against 3.9605. When the cost weight is raised to four, the small candidate wins. The latency and data-boundary weights do nothing other than enlarge the medium candidate.

The same three candidates, the same four numbers, three different decisions. The only thing that changes is which criterion counts for how much. For this reason the question “which model is better” has no measure; the question that has a measure is “with these four numbers and these weights, which candidate wins”. A justification for a choice, as long as it does not write its weights, is not a justification but a preference.

The normalization rule is itself a decision, and it does not show in the table. The rule used here scales to the range: in each criterion the worst candidate gets 0 and the best gets 1. This rule does not look at how large the difference in between is — on hit rate the 0.0833 difference between small and medium is divided by the 0.2917 width of the range and rises to 0.2857, that is, a small absolute difference turns into a large normalized difference. Another rule would have given other numbers. Because the Environment Selection lesson measured that this can turn the choice, a single rule is used here and the fact that it is used is written; when it is not written, the table looks more objective than it is.

The Threshold and the Choice’s Own Uncertainty

def winner(N, w):
    p = [sum(w[j] * N[k][i] for j, (_, k, _) in enumerate(CRITERIA)) for i in range(3)]
    return CANDIDATES[p.index(max(p))][0]


print(f"{'swept criterion':<20}{'weight 1':<10}{'first change':<14}new winner")
for j, (name, _, _) in enumerate(CRITERIA):
    before, threshold, new = winner(N, (1, 1, 1, 1)), "-", "-"
    for w in range(2, 13):
        k = winner(N, tuple(w if i == j else 1 for i in range(4)))
        if k != before and threshold == "-":
            threshold, new = w, k
    print(f"{name:<20}{before:<10}{str(threshold):<14}{new}")
S2 = [measure(c, SEED + 1) for c in CANDIDATES]
N2 = {k: normalize(S2, k, d) for _, k, d in CRITERIA}
print(f"{'weight set':<24}{'seed 20260218':<16}seed 20260219")
for name, w in WEIGHTS:
    print(f"{name:<24}{winner(N, w):<16}{winner(N2, w)}")

swept criterion     weight 1  first change  new winner
hit rate            medium    4             large
equivalent tokens   medium    2             small
steps               medium    -             -
requests crossing   medium    -             -
weight set              seed 20260218   seed 20260219
balanced                medium          medium
hit-rate weighted       large           medium
cost weighted           small           small
latency weighted        medium          medium
data-boundary weighted  medium          medium

The thresholds are asymmetric. When the cost weight is 2, the choice moves from medium to small; the hit-rate weight does not bring the large candidate ahead until it reaches 4. Even when the latency and data-boundary weights are swept up to twelve, they do not change the choice, because on these two criteria the winner is already medium and raising the weight does nothing other than reinforce the winner. Enlarging the weight of a criterion never brings ahead a candidate that is not ahead on that criterion.

The last table is the most uncomfortable number of this lesson. When the same computation is repeated with seed 20260219, the winner of the hit-rate weighted set turns from large back to medium. No criterion changed, no weight changed; only the sampling seed of the simulator changed, and the hit rate of the large candidate fell from 0.7917 to 0.7083. A decision won by a hair, when it is built on a component that changes from run to run, becomes dependent on the run itself. In three of the four criteria the choice is solid; in the fourth it is not, and this is something a comparison measured with a single run could never see.

A practical rule follows from this. A selection table should write not only the winner but also the winning margin. Rows with a wide margin stand even if the weights wobble a little or the measurement drifts a little; those with a thin margin do not. The margin of 0.1450 between 3.1033 and 2.9583 at equal weights and the margin of 0.0395 between 4.0000 and 3.9605 in the hit-rate weighted set stand in the same table but do not say the same thing. In a row with a thin margin the right behavior is not to declare the winner but to run both candidates again by enlarging the measurement, or to tie the decision to one more criterion.

Summary

• When three candidates are counted on four criteria the winner splits four ways: large on hit rate with 0.7917, small on equivalent tokens with 89.5, medium on steps with 7.00, small and medium on requests crossing the organization boundary with 0/24.
• The small candidate’s scaffolding raises the hit rate to 0.5000 while taking the number of calls to 1.4583 and the steps to 7.29; the gain and the cost stand in the same table.
• Across five weight sets the winner is medium three times, large once and small once; at equal weights the margin is thin, like 3.1033 against 2.9583.
• The thresholds are asymmetric: a cost weight of 2 turns the choice from medium to small, and a hit-rate weight of 4 turns it from medium to large; the latency and data-boundary weights do not change the choice up to twelve.
• With seed 20260219 the winner of the hit-rate weighted set turns from large back to medium; a choice given by a hair becomes dependent on the run.

Next Step

In this lesson cost was counted as a criterion, but it was never used as a limit. The winning candidate was chosen freely under every weight set; yet in a real system the token budget is an upper bound and imposes not the choice but the design. Even with the same task, the same model and the same hit rate, how many tokens are spent per request varies greatly with how the prompt is written. The next lesson compares four scaffoldings as tokens and hit rate — a long prompt, a short prompt with examples, summarize and send, and a cache — and shows which design becomes mandatory when the budget narrows. That lesson is also the close of the course.