Token-Based Cost

In the previous lesson the token was counted as a criterion but was never used as a limit. In a real system the budget is an upper bound and imposes not the choice but the design. Even with the same task, the same model and the same hit rate, the tokens spent per request change several times over depending on how the prompt is written.

This lesson compares four scaffoldings on the same request stream, as tokens and hit rate. The unit is the token; no currency is written, because the number of tokens is measurable and lasting, while a price is neither measurable nor lasting.

• LM71. The request stream is 96 fictional support requests and contains repetition: 24 requests are drawn so that their frequencies are unequal. The requests and the classes are fictional, the seed is 20260218.
• LM72. The measure is twofold: tokens per request (small is good) and hit rate (large is good). The number of model calls is written separately.
• LM73. The four scaffoldings are like a ladder, each row adds something to the one before: long prompt, short prompt with examples, summarize and send, cache. The fourth row, the hard summary, is a harder setting of the third row and is measured to show the limit.
• LM74. Summarize and send trims the request before putting it into the prompt. The trimming is done before the model, on the organization’s own side, and makes the model charge no tokens at all.
• LM75. The cache stores the response by request. When the same request comes a second time the model is not called at all; if the stored response is correct it is repeated as correct, if it is wrong it is repeated as wrong.
• LM76. Uncertainty is measured with two runs; the second run is taken with seed 20260219.

Tokens and Hit Rate of the Four Scaffoldings

# SHARED -- FICTIONAL. The simulator does not call a real endpoint. The cost unit is the TOKEN;
# no currency is written. The request stream is 96 fictional support requests with repetition.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def tokenize(text):
    """Fictional tokenizer: letter pieces are split into fours, digit pieces into twos."""
    out, buffer = [], ""
    for ch in text + " ":
        if ch.isalnum():
            buffer += ch
            continue
        if buffer:
            n = 2 if buffer.isdigit() else 4
            out += [buffer[i:i + n] for i in range(0, len(buffer), n)]
            buffer = ""
        if ch.strip():
            out.append(ch)
    return out


MAIN = [("meter not read estimated invoice issued", "reading"),
        ("invoice amount doubled compared to last month", "payment"),
        ("water pressure dropped during the night", "water"),
        ("planned outage notice did not arrive", "outage"),
        ("subscription transfer was not applied to meter", "subscription")]
EXTRA = ["meter number may have been changed", "address information looks old",
         "same problem did not occur last month", "cannot reach the meter door closed",
         "outage lasted longer than expected", "no automatic payment instruction on meter"]
LEXICON = {"outage": ["outa", "noti"], "reading": ["read", "meas", "esti"],
           "water": ["pres", "drop", "tank", "pipe"],
           "payment": ["invo", "bill", "paym", "amou"],
           "subscription": ["subs", "addr", "tran", "cont"]}
CLASSES = list(LEXICON)
REQUESTS = []
for i in range(24):
    r = rng(SEED + 31 * i)
    a, b = 100000 + int(r() * 380000), 100000 + int(r() * 380000)
    REQUESTS.append({"id": 4100 + i, "label": MAIN[i % 5][1],
                     "text": f"subscriber {4100 + i}: {MAIN[i % 5][0]}, {EXTRA[(5 * i + 2) % 6]}."
                             f" main meter {a} sub meter {b}."})
r = rng(SEED + 9091)
STREAM = [int(r() ** 1.7 * 24) for _ in range(96)]   # fictional request stream with repetition
# (name, instruction tokens, example tokens, input trim, cache)
SCAFFOLDINGS = [("long prompt", 118, 0, 0, False), ("short prompt + example", 16, 46, 0, False),
                ("summarize and send", 16, 46, 12, False), ("hard summary", 16, 46, 6, False),
                ("cache", 16, 46, 12, True)]


def call(t, trim, order, seed):
    """Simulator: uses the tokens visible in the prompt; trimming shortens the visible part."""
    tokens = tokenize(t["text"])
    tokens = tokens[:trim] if trim else tokens
    known = {s: set(o[:4]) for s, o in LEXICON.items()}
    p = sorted(((sum(b in known[s] for b in tokens), s) for s in CLASSES), key=lambda x: (-x[0], x[1]))
    if p[0][0] == 0 or p[0][0] == p[1][0]:
        return "ambiguous"
    return p[1][1] if rng(seed + 37 * order + 7 * t["id"])() < 0.14 else p[0][1]


def measure(scaffolding, seed=SEED):
    name, instruction, example, trim, cached = scaffolding
    cache, correct, tokens, calls = {}, 0, 0, 0
    for order, i in enumerate(STREAM):
        t = REQUESTS[i]
        if cached and i in cache:
            correct += cache[i] == t["label"]
            continue                            # the request never goes to the model
        n = len(tokenize(t["text"]))
        y = call(t, trim, order, seed)
        tokens += instruction + example + (min(n, trim) if trim else n)
        calls += 1
        cache[i] = y
        correct += y == t["label"]
    return correct / len(STREAM), tokens / len(STREAM), calls, tokens


print(f"fictional request stream {len(STREAM)} requests, unique requests {len(set(STREAM))}/24")
print("scaffolding             hit rate  hit rate B  tokens/request  model calls  total tokens")
for s in SCAFFOLDINGS:
    h, b, c, total = measure(s)
    h2 = measure(s, SEED + 1)[0]
    print(f"{s[0]:<24}{h:<10.4f}{h2:<12.4f}{b:<16.1f}{f'{c}/96':<13}{total}")

fictional request stream 96 requests, unique requests 23/24
scaffolding             hit rate  hit rate B  tokens/request  model calls  total tokens
long prompt             0.7083    0.7396      159.8           96/96        15341
short prompt + example  0.7083    0.7396      103.8           96/96        9965
summarize and send      0.8021    0.8438      74.0            96/96        7104
hard summary            0.0833    0.0833      68.0            96/96        6528
cache                   0.7500    0.9583      17.7            23/96        1702

The first two rows give the same hit rate: 0.7083, and in the second run 0.7396. The only difference between them is that two examples are put in place of a long instruction, with a short instruction. The difference in cost is 56.0 tokens per request, 5,376 tokens in total. This is the easiest gain in the table, and it makes the hit rate pay nothing. The long prompt looks like a good prompt for as long as it is not measured.

The third row goes in an unexpected direction: summarize and send brings the tokens down to 74.0 while raising the hit rate to 0.8021. The reason is in the fiction and can be read — the trimming drops the extra clause of the request, and the extra clause carried the pieces that blurred the class evidence. This is not a general law, it is a property of this data set. Indeed the fourth row shows the limit: when the trimming is hardened to six tokens the hit rate collapses to 0.0833, and in return only 6.0 tokens are saved. A summary is a setting, and the good side of the setting and its cliff are very close to each other.

The last row carries the largest gain in the table: 17.7 tokens per request, model calls 23/96. Seventy-three of the ninety-six requests never go to the model. The hit rate is 0.7500, below the third row, but the two-run columns must be looked at: the same cache gives 0.9583 with the second seed. The cache neither lowers nor raises the hit rate; it freezes it. Whatever the first response to a request turned out to be, all the repetitions of that request receive that response. A cache placed in front of an uncertain component does not remove the uncertainty; it reduces it to a single draw and makes the result permanent.

The Design the Budget Imposes

RESULTS = [(s[0],) + measure(s) for s in SCAFFOLDINGS]
print("budget  per request  scaffoldings that fit  design imposed by the budget  hit rate")
for budget in (16000, 10000, 7000, 2000, 1500):
    fitting = [s for s in RESULTS if s[4] <= budget]
    best = max(fitting, key=lambda s: s[1]) if fitting else None
    print(f"{budget:<8}{budget / 96:<13.1f}{f'{len(fitting)}/5':<23}"
          f"{(best[0] if best else 'none'):<30}{(f'{best[1]:.4f}' if best else '-')}")
print("scaffolding without a written cost: long prompt and short prompt + example give the same hit rate,"
      " difference 5376 tokens")

budget  per request  scaffoldings that fit  design imposed by the budget  hit rate
16000   166.7        5/5                    summarize and send            0.8021
10000   104.2        4/5                    summarize and send            0.8021
7000    72.9         2/5                    cache                         0.7500
2000    20.8         1/5                    cache                         0.7500
1500    15.6         0/5                    none                          -
scaffolding without a written cost: long prompt and short prompt + example give the same hit rate, difference 5376 tokens

When the budget is wide the design is chosen: at 16,000 tokens all five of the five scaffoldings fit and the one that gives the highest hit rate is selected. When the budget drops to 7,000 the design is imposed: only two scaffoldings fit, and the best is not 0.8021 but 0.7500. The 0.0521 in between is not an engineering error, it is the price of the budget. At 1,500 tokens no scaffolding fits; at that point what to do is not to write a better prompt but either to enlarge the budget or to keep part of the requests from passing through the model at all.

The order in which this table is read also matters. As the budget narrows, the first scaffolding to fall is the most expensive one, not the worst one. The long prompt, at 15,341 tokens, is the first to leave the list, and its hit rate is exactly the same as the short prompt’s; so the budget first cuts the cost that buys nothing. This is also the first step in making a system cheaper. A scaffolding whose cost is not written cannot be defended, because it has not been measured.

Summary

• The long prompt and the short prompt plus examples give the same hit rate (0.7083); the difference is 56.0 per request and 5,376 tokens in total, and it makes the hit rate pay nothing.
• Summarize and send brings the tokens down to 74.0 while raising the hit rate to 0.8021, but when the trimming is hardened to six tokens the hit rate collapses to 0.0833 and the return is only 6.0 tokens.
• The cache is the cheapest at 17.7 tokens per request and never sends 73 of the 96 requests to the model; it neither lowers nor raises the hit rate, it reduces it to a single draw and freezes it (0.7500 and 0.9583).
• When the budget drops to 7,000 tokens two of the five scaffoldings fit and the design is imposed rather than chosen; the best reachable hit rate falls from 0.8021 to 0.7500.
• The first scaffolding to leave as the budget narrows is the most expensive one and it contributes nothing to the hit rate; the first step in making a system cheaper is removing the scaffoldings whose gain has not been measured.

Course Wrap-Up

The accounting of fourteen lessons is in a single table, and every number was read from its own lesson. The rows do not share a common unit; the second column states by which measure each row is read. The difference is given in a single sign convention: the number of the scaffolding minus the number of the bare call, in parentheses in the third column.

This is the rule of the course: a given component is measured from its surface; the number of a component whose inside cannot be seen is written on its surface. None of the numbers in the table came from inside the model. All of them were taken by counting where a request went in, where a response came out, and how many tokens, how many calls and how many steps were spent in between. The model’s architecture, its weights and the data it was trained on appear nowhere in the table, because these were not measurable quantities in this course; everything that was measurable was on the surface.

The rule of the curriculum can also be read from here: the number of an artificial intelligence system is not the number of the model but the number of the scaffolding; a scaffolding whose cost is not written counts as unmeasured. This is why the fourth column stands in the table and is empty in no row.

The second claim of the curriculum can also be read by name from the same table: adding scaffolding does not always improve. In Field 02 the retry moved the number by 0.0000 and added one call and 51 tokens. In Field 03 offering options raised the visible change from 0.3056 to 0.4306. In Language 01 the 0.30 threshold lowered the hit rate from 0.9722 to 0.8611. In Language 03 fine-tuning fixed the domain text while bringing the general text down from 0.3186 to 0.2692. In Language 05 summarizing brought the prompt down to 100.6 tokens but, because it dropped the step that was asked about, collapsed the full-response rate from 0.3833 to 0.0000. In Language 06 the three-run majority spent 99.9 tokens and changed nothing in counting and exact arithmetic. In Language 07 the abstention option dropped the hit rate 0.0417 below the baseline. In Language 10 the hard summary collapsed the hit rate from 0.8021 to 0.0833. Eight rows, a single rule: the cost of a scaffolding is certain, its gain is not.

In fourteen lessons the surface of the model was measured and the cost was written next to every number. The only thing not asked is what the model was told. In the whole table the prompt was either held fixed or mentioned by its length; what is written in it, in which order, where an example is placed and how the instruction is built was never a variable. How much the number moves when the same model is given a different text on the same task, and that the text itself is a design object, was not measured. The next course, Prompt Engineering, opens exactly this variable.