Capabilities and Limits

The previous lesson measured the context window as a limit and showed that how to fill the window is a decision. The question of how many tokens the truncation decisions save and how much they lower the hit rate was answered. What was not asked comes earlier: whatever is put into the window, what the model does well was never measured. If a task is not going well, is the fault in the prompt, in the window, or in the task itself?

This lesson runs five task classes on the same data with the same bare call and writes the hit rate per class. In two classes the number collapses, and the source of the collapse is not in the prompt, it is at the token level.

• LM46. The fictional request set is 24 short Turkish support requests. The requests are generated from the combination of five main sentences and six extra sentences, and two meter readings stand at the end of each request. The texts, the classes and the readings are fictional.
• LM47. The model is a simulator; no real endpoint is called. The simulator is written with the standard library, its seed is 20260218, and it produces its response only from the token sequence.
• LM48. The tokenizer is fictional and carries on the measure set up in the first lesson of this topic: letter pieces are split into fours, digit pieces into twos.
• LM49. Five task classes are measured: summarization, classification, ranking, counting and exact arithmetic. The measure of each class is the hit rate: in how many of the twenty-four requests the response is correct.
• LM50. The cost is written in three units: input tokens per request, number of calls and latency class. Raw time is not written.
• LM51. The uncertainty is measured with two runs. The same task set is run with seed 20260218 and seed 20260219; in addition, the number of requests whose response changed between the two runs is printed.

Bare Call on Five Task Classes

# COMMON -- FICTIONAL. The simulator below does not call a real endpoint; it is written with the
# standard library and carries the measurable surface of a language model.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def tokenize(text):
    """Fictional tokenizer: letter pieces are split into fours, digit pieces into twos."""
    out, buffer = [], ""
    for ch in text + " ":
        if ch.isalnum():
            buffer += ch
            continue
        if buffer:
            n = 2 if buffer.isdigit() else 4
            out += [buffer[i:i + n] for i in range(0, len(buffer), n)]
            buffer = ""
        if ch.strip():
            out.append(ch)
    return out


MAIN = [  # (main sentence, true class)
    ("sayac okunmadi tahmini fatura cikti", "olcum"),
    ("fatura tutari gecen aya gore iki kat cikti", "odeme"),
    ("gece boyunca su basinci dustu", "tesisat"),
    ("planli kesinti bildirimi ulasmadi", "kesinti"),
    ("abonelik devri sayaca islenmedi", "sozlesme"),
]
EXTRA = ["sayacin numarasi degistirilmis olabilir", "adres bilgisi eski gorunuyor",
         "onceki ay ayni sorun yasanmadi", "sayaca ulasilamiyor kapi kapali",
         "kesinti suresi beklenenden uzun surdu", "otomatik talimat sayacta yok"]
LEXICON = {  # class -> pieces the simulator recognizes and fictional priority
    "kesinti": ({"kesi", "bild"}, 1), "olcum": ({"okun", "olcu", "tahm"}, 2),
    "tesisat": ({"basi", "vana", "tesi", "dust"}, 3),
    "odeme": ({"fatu", "odem", "tuta", "tali"}, 4),
    "sozlesme": ({"abon", "devr", "sozl", "adre"}, 5),
}
CLASSES = list(LEXICON)
TASKS = {"summarization": 11, "classification": 23, "ranking": 41, "counting": 59,
         "exact_arithmetic": 71}
SAMPLES = {}
REQUESTS = []
for i in range(24):
    r = rng(SEED + 31 * i)
    a, b = 100000 + int(r() * 380000), 100000 + int(r() * 380000)
    REQUESTS.append({"id": 4100 + i, "label": MAIN[i % 5][1], "a": a, "b": b,
                     "main": MAIN[i % 5][0], "extra": EXTRA[(5 * i + 2) % 6],
                     "text": f"{MAIN[i % 5][0]}, {EXTRA[(5 * i + 2) % 6]}."
                             f" ana sayac {a} ara sayac {b}."})


def draw(task, seed, t):
    """Sampling table: one stream per task and seed, one value per request."""
    if (task, seed) not in SAMPLES:
        r = rng(seed + 1009 * TASKS[task])
        SAMPLES[(task, seed)] = [r() for _ in range(24)]
    return SAMPLES[(task, seed)][t["id"] - 4100]


def call(task, t, seed, sampling=True):
    """Simulator: produces the response only from the token sequence. In tasks that choose
    among candidates, sampling can bring the second candidate; counting and exact arithmetic
    produce no candidates, they follow a single path over the token sequence."""
    tokens = tokenize(t["text"])
    if task == "counting":
        return tokens.count("saya")
    if task == "exact_arithmetic":
        p = [(str(t["a"])[k:k + 2], str(t["b"])[k:k + 2]) for k in range(0, 6, 2)]
        return int("".join(f"{(int(x) + int(y)) % 100:02d}" for x, y in p))
    r = draw(task, seed, t)
    if task == "summarization":                   # which sentence enters the summary
        sentence = sorted([t["main"], t["extra"]], key=lambda c: -len(tokenize(c)))
        return sentence[1] if sampling and r < 0.14 else sentence[0]
    ranked = sorted(CLASSES, key=lambda s: (-sum(b in LEXICON[s][0] for b in tokens), s))
    return ranked[1] if sampling and r < 0.14 else ranked[0]


def measure(seed, sampling=True):
    d, ordered = {}, 0
    d["summarization"] = sum(call("summarization", t, seed, sampling) == t["main"]
                             for t in REQUESTS) / 24
    d["classification"] = sum(call("classification", t, seed, sampling) == t["label"]
                              for t in REQUESTS) / 24
    for g in range(24):
        three = [REQUESTS[g], REQUESTS[(g + 7) % 24], REQUESTS[(g + 13) % 24]]
        truth = [t["id"] for t in sorted(three, key=lambda t: (LEXICON[t["label"]][1], t["id"]))]
        predicted = [t["id"] for t in sorted(three, key=lambda t: (
            LEXICON[call("ranking", t, seed, sampling)][1], t["id"]))]
        ordered += truth == predicted
    d["ranking"] = ordered / 24
    d["counting"] = sum(call("counting", t, seed) == t["text"].replace(",", " ")
                        .replace(".", " ").split().count("sayac") for t in REQUESTS) / 24
    d["exact_arithmetic"] = sum(call("exact_arithmetic", t, seed) == t["a"] + t["b"]
                                for t in REQUESTS) / 24
    return d


RUN_A, RUN_B = measure(SEED), measure(SEED + 1)
MEAN_TOKENS = sum(len(tokenize(t["text"])) for t in REQUESTS) / 24
print(f"fictional task set: {len(REQUESTS)} support requests, five task classes")
print(f"input tokens per request {MEAN_TOKENS:.1f}, calls 1, latency class single call")
print(f"{'task':<22}{'seed 20260218':<16}{'seed 20260219':<16}changed requests")
for g in TASKS:
    diff = sum(str(call(g, t, SEED)) != str(call(g, t, SEED + 1)) for t in REQUESTS)
    print(f"{g:<22}{RUN_A[g]:<16.4f}{RUN_B[g]:<16.4f}{diff:>2}/24")

fictional task set: 24 support requests, five task classes
input tokens per request 33.3, calls 1, latency class single call
task                  seed 20260218   seed 20260219   changed requests
summarization         0.7500          0.6667           4/24
classification        0.8750          0.8750           2/24
ranking               0.6250          0.4583          11/24
counting              0.4167          0.4167           0/24
exact_arithmetic      0.2917          0.2917           0/24

The number falls from class to class: classification 0.8750, summarization 0.7500, ranking 0.6250, counting 0.4167, exact arithmetic 0.2917. The same model, the same request, the same token count. The only thing that changes is what the task asks for.

Ranking giving the lowest number among the upper three rows has its own reason. Ranking is not a single decision but the combination of three: the simulator classifies each request separately and orders the triple by the priority of the classes. Because a mistake on a single request spoils the whole order, the hit rate of 0.8750 per class falls to 0.6250 per triple. The number of composite tasks is always lower than the number of their components, and this is a rule that will continue throughout this lesson.

The upper three classes also share something: all of them are a selection task. Summarization chooses one of two sentences, classification one of five labels, ranking the order of three requests. In selection tasks the token surface carries enough information, because the correct answer is already somewhere inside the text. The lower two classes are generation tasks: the correct answer is not found inside the text, it has to be computed from the text. What shows the distinction, however, is not the measure itself but how the measure is taken.

The last column says this alone. In the first three classes there are requests whose response changes when the seed changes: 11/24 in ranking, 4/24 in summarization, 2/24 in classification. The hit rate of classification comes out the same in the two runs but not on the same requests: two requests changed their response, one improved and one got worse. In the last two classes the difference is 0/24. Counting and exact arithmetic give exactly the same wrong answer in both runs. This is not sampling noise.

The Token-Level Source of Counting and Exact Arithmetic

def inflected(m):
    return [w for w in m.replace(",", " ").replace(".", " ").split()
            if w.startswith("sayac") and w != "sayac"]


t = REQUESTS[4]
print("request:", t["main"] + ",", t["extra"])
print("tokens:", " ".join(tokenize(t["main"])))
truth = t["text"].replace(",", " ").replace(".", " ").split().count("sayac")
print(f"sayac occurrences: true {truth}, simulator {call('counting', t, SEED)},"
      f" inflected form {inflected(t['text'])}")
print(f"addition: {t['a']} + {t['b']} = {t['a'] + t['b']},"
      f" simulator {call('exact_arithmetic', t, SEED)}")
carries = sum(any(int(str(x["a"])[k:k + 2]) + int(str(x["b"])[k:k + 2]) > 99 for k in (2, 4))
              for x in REQUESTS)
inflected_count = sum(bool(inflected(x["text"])) for x in REQUESTS)
print(f"requests with a carry across a piece boundary {carries}/24 -> remaining {24 - carries}/24")
print(f"requests with an inflected sayac form {inflected_count}/24"
      f" -> remaining {24 - inflected_count}/24")

request: abonelik devri sayaca islenmedi, kesinti suresi beklenenden uzun surdu
tokens: abon elik devr i saya ca isle nmed i
sayac occurrences: true 2, simulator 3, inflected form ['sayaca']
addition: 139039 + 302204 = 441243, simulator 431243
requests with a carry across a piece boundary 17/24 -> remaining 7/24
requests with an inflected sayac form 14/24 -> remaining 10/24

Counting first. The model does not see words in the text; it sees pieces. The word sayaca (to the meter) is split into the pieces saya and ca, and the word sayac (meter) also starts with saya. Because the two share the same beginning in the token sequence, the model counts the inflected form as well and writes 3 instead of 2. The agglutinative structure of Turkish makes this a rule, not an exception: requests carrying an inflected form of sayac are 14/24, those not carrying one 10/24. The counting hit rate is 0.4167, that is exactly 10/24. The model is wrong not because it cannot count but because it cannot see the unit it is to count.

Exact arithmetic is the number side of the same flaw. The reading 139039 is split into the pieces 13, 90, 39. The model adds the pieces pairwise and keeps each piece at two digits: when 90 + 22 comes to 112, the third digit, that is the carry, cannot pass to the next piece because that piece has already been written. The number that comes out is 431243, the correct one is 441243: the difference is exactly the lost carry. Requests carrying a carry across a piece boundary are 17/24, those not carrying one 7/24. The exact-arithmetic hit rate is 0.2917, that is exactly 7/24. Both numbers come not from the data but from tokenization; changing the seed does not move them.

Adding Scaffolding to the Collapsing Class

TRUTH = {"classification": lambda t: t["label"], "exact_arithmetic": lambda t: t["a"] + t["b"],
         "counting": lambda t: t["text"].replace(",", " ").replace(".", " ")
         .split().count("sayac")}


def majority(task):                         # scaffolding 1: three runs, majority response
    s = 0
    for t in REQUESTS:
        y = [call(task, t, k) for k in (SEED, SEED + 1, SEED + 2)]
        s += max(y, key=y.count) == TRUTH[task](t)
    return s / 24


def with_procedure(task, t):
    """Scaffolding 2: the model extracts the value, a procedure outside the tokens does the calculation."""
    if task == "exact_arithmetic":
        numbers, o, tokens = [], 0, tokenize(t["text"])
        while o < len(tokens):                  # adjacent digit pieces form one number
            p = ""
            while o < len(tokens) and tokens[o].isdigit():
                p, o = p + tokens[o], o + 1
            o += 1 if not p else 0
            if p:
                numbers.append(int(p))
        return sum(numbers[:2])
    return sum(w == "sayac" for w in t["text"].replace(",", " ").replace(".", " ").split())


print(f"{'task':<18}{'bare call':<11}{'majority':<10}procedure")
for g in ("classification", "counting", "exact_arithmetic"):
    y = ("-" if g == "classification" else
         f"{sum(with_procedure(g, t) == TRUTH[g](t) for t in REQUESTS) / 24:.4f}")
    print(f"{g:<18}{measure(SEED)[g]:<11.4f}{majority(g):<10.4f}{y}")
extra_tokens = sum(len(tokenize(str(TRUTH['exact_arithmetic'](t)))) for t in REQUESTS) / 24
print(f"cost -- bare call: {MEAN_TOKENS:.1f} tokens, 1 call, latency class single call")
print(f"cost -- majority: {3 * MEAN_TOKENS:.1f} tokens, 3 calls, latency class parallel")
print(f"cost -- procedure: {2 * MEAN_TOKENS + extra_tokens:.1f} tokens, 2 calls, latency class chain")

task              bare call  majority  procedure
classification    0.8750     0.9167    -
counting          0.4167     0.4167    1.0000
exact_arithmetic  0.2917     0.2917    1.0000
cost -- bare call: 33.3 tokens, 1 call, latency class single call
cost -- majority: 99.9 tokens, 3 calls, latency class parallel
cost -- procedure: 69.6 tokens, 2 calls, latency class chain

The majority scaffolding calls the same request with three seeds and takes the most frequent response. It works in classification: the number 0.8750 rises to 0.9167, because the error there comes from sampling and two of the three runs hold the correct answer. In counting and exact arithmetic, however, the number does not budge: 0.4167 and 0.2917 stay exactly the same. The cost is real and paid up front: the number of calls goes to three, the tokens to 99.9. A statistical scaffolding placed on top of a structural error does nothing but repeat the error three times.

The second scaffolding moves the work outside the token layer: the model only extracts the numbers, and a procedure does the adding and the counting. The hit rate is 1.0000 in both classes. The cost is two calls, 69.6 tokens and the chain latency class — because the second call has to wait for the result of the first, while the three calls of the majority scaffolding can go in parallel. The cheaper scaffolding is the one that pays off: majority gives nothing with 99.9 tokens, the procedure corrects the class entirely with 69.6 tokens. Without writing the cost these two scaffoldings could not have been told apart.

Summary

• When five task classes are measured with the bare call on the same fictional request set, the hit rate falls from 0.8750 to 0.2917: classification 0.8750, summarization 0.7500, ranking 0.6250, counting 0.4167, exact arithmetic 0.2917.
• When the seed changes, there are requests whose response changes in the first three classes (11/24, 4/24, 2/24) and 0/24 in the last two; the error of counting and exact arithmetic is not sampling noise.
• The counting hit rate equals exactly the 10/24 requests that carry no inflected form, the exact-arithmetic hit rate equals exactly the 7/24 requests that carry no carry across a piece boundary; both numbers come from tokenization.
• The three-run majority scaffolding carries classification to 0.9167 but changes nothing in counting and exact arithmetic; its cost is 99.9 tokens and three calls.
• The procedure scaffolding that takes the calculation out of the token layer carries both classes to 1.0000; its cost is 69.6 tokens, two calls and the chain latency class.

Next Step

The wrong answers measured in this lesson were honest in a way: when the model could not count it wrote a wrong number, and the wrongness was visible in a single line. The next lesson takes up a harder kind of wrong answer. To a question, even when it holds no basis whatsoever, the model produces a fluent and confident response; the form of the response is the same as the correct one, and its wrongness cannot be understood by looking at the response itself. The measure of this is not the hit rate but the fabrication rate: how many claims in the response rest on a document. The same lesson shows the second claim of the curriculum for the first time: one of the three scaffoldings added to reduce fabrication really lowers fabrication, and lowers the hit rate too, and this is not hidden.