Tokens

The Field and Roles topic looked at the model from the outside across four lessons: who builds what, how many settings the training approach and the using approach have in hand, what the same input gives in two runs, and which claim could be tied to a test. In all four the model was a box and the lid of the box was never opened. What is inside, which unit it works with and where its limit comes from were never asked.

This topic opens with that question and starts from the very bottom. Sending text to a language model is not really sending text: the text is first split into pieces, each piece is turned into a number, and the model sees only that sequence of numbers. This piece is called a token. A token is not a technical detail; it is the unit of cost of this curriculum: the window is measured in tokens, the fee is counted in tokens, latency grows with the number of generated tokens. The rule of the course works here for the first time — a given component is measured from its surface, and the surface this lesson measures is the token count.

• LM1. The support tickets are fictional: they are generated inside the lesson, seed 20260218, and come from no source system. The set is 300 short Turkish tickets.
• LM2. The model is a simulator: it is written inside the lesson with the standard library, and no real model endpoint is called. The surface it carries is tokenization, token counting, the context window, the stop condition, the maximum output length and seed-dependent uncertainty.
• LM3. The tokenizer is fictional too: a vocabulary of 177 pieces and leftmost longest match. The tokenization procedure, the vocabulary and the out-of-vocabulary concepts were measured in the Natural Language Processing and Computer Vision course; the procedure is not repeated here, it is only read as a unit of cost.
• LM4. A sequence with no counterpart in the vocabulary falls to single characters. In this simulator there is no out-of-vocabulary case; the cost of the unknown is paid as a token count.
• LM5. Every ticket has a field triple: object, status, request. The triple is known from the fiction itself; the reader procedure takes the first matching root in the token sequence.
• LM6. The measure is the same on every row: tokens per ticket (smaller is better) and field hit rate (larger is better).
• LM7. In this lesson the bare call is sending the ticket as it is; the scaffolding is every operation applied to the text before it is sent.
• LM8. The input token count is independent of the run: the same text gives the same number in every run. The output length is not independent.

The Input Unit of Text for the Model

The simulator’s tokenizer carries a small vocabulary of pieces: the frequent roots of the field, frequent suffixes and single characters. A word is scanned from the left, at each step the longest piece found in the vocabulary is taken, and if none is found a single character is taken. Punctuation is a token on its own.

# SIMULATOR — in this course the model is a simulator written in the lesson with
# the standard library; no real model endpoint is called. The support tickets
# are FICTIONAL and generated in the lesson, seed 20260218.
SEED, M32, TICKETS = 20260218, 0xFFFFFFFF, 300

ROOTS = """sayac kapak muhur vana abone adres fatura tarife odeme borc ariza kod okuma
olcum servis ekip ziyaret kayit talep destek hat bakim birim gun ay yil saat numara
bilgi islem sistem hata sonuc deger endeks sozlesme kampanya indirim gecikme faiz
itiraz basvuru sikayet cevap bekle gonder geri ac kapa oku yaz bul bak gel ver yap
don calis bozuk kirik eksik normal yuksek dusuk yeni eski acik kapali lutfen merhaba
tesekkur iyi rica eder mumkun tekrar hala durum konu ile icin ve bir bu ne var yok
ama kadar sonra once ben biz siz""".split()
SUFFIXES = """lari leri lar ler nin nun den dan de da te ta iniz unuz imiz umuz in un im um
si su li lu di du ti tu miz niz yor mis dir tir cek cak ki ya ye yi yu ma me ni nu
la le i u e a""".split()
PIECES = set(ROOTS) | set(SUFFIXES) | set("abcdefghijklmnopqrstuvwxyz0123456789")
PUNCT = ".,:;?-/"


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def split_pieces(s):
    p, i = [], 0
    while i < len(s):
        for j in range(min(len(s), i + 9), i, -1):
            if s[i:j] in PIECES:
                p.append(s[i:j])
                i = j
                break
        else:
            p.append(s[i])
            i += 1
    return p


def tokenize(text):
    out, cur = [], ""
    for c in text.lower():
        if c.isalnum():
            cur += c
        else:
            if cur:
                out += split_pieces(cur)
                cur = ""
            if c in PUNCT:
                out.append(c)
    return out + (split_pieces(cur) if cur else [])


OBJECTS = {"fatura": ("faturamizin", ["yuksek", "yuksek", "eksik"]),
           "tarife": ("tarifemizin", ["yuksek", "eski", "yuksek"]),
           "endeks": ("endeksimizin", ["yuksek", "dusuk", "eksik"]),
           "sayac": ("sayacimizin", ["bozuk", "kirik", "eski"]),
           "kapak": ("kapaginin", ["kirik", "eksik", "acik"]),
           "vana": ("vananin", ["kapali", "bozuk", "acik"]),
           "hat": ("hattimizin", ["kapali", "bozuk", "eski"]),
           "odeme": ("odememizin", ["eksik", "gecikmeli", "yuksek"])}
OBJECT_WEIGHTS = [0.20, 0.14, 0.12, 0.14, 0.10, 0.09, 0.11, 0.10]
REQUESTS = {"gonder": "ekip gonderin", "ac": "kayit acin",
            "ver": "bilgi verin", "don": "geri donun"}
TEMPLATES = ["{a} {n} durumu {d} ve bu konuda hala bir cevap yok. {i} lutfen. {k}.",
             "{a} {n} durumu {d} ve tekrar yazdim ama sonuc yok. {i} lutfen. {k}.",
             "{a} {n} durumu {d} ve bu ay ikinci basvurum. {i} lutfen. {k}."]
OPENINGS = ["Merhaba,", "Iyi gunler,", "Merhaba,", "Iyi gunler,"]
CLOSINGS = ["Tesekkur ederim", "Rica ederim", "Tesekkur ederim", "Iyi gunler"]
OBJECT_NAMES, REQUEST_ROOTS = list(OBJECTS), list(REQUESTS)

DATA = []
for i in range(TICKETS):
    r = rng(SEED + i)
    n = OBJECT_NAMES[pick(r(), OBJECT_WEIGHTS)]
    long_name, statuses = OBJECTS[n]
    d = statuses[int(r() * 3)]
    k = REQUEST_ROOTS[int(r() * 4)]
    text = TEMPLATES[int(r() * 3)].format(a=OPENINGS[int(r() * 4)], n=long_name, d=d,
                                          i=REQUESTS[k], k=CLOSINGS[int(r() * 4)])
    DATA.append({"text": text, "field": (n, tokenize(d)[0], k)})

print(f"{TICKETS} fictional support tickets, simulator vocabulary {len(PIECES)} pieces")
for i in (0, 2, 7):
    b = tokenize(DATA[i]["text"])
    print(f"\n{DATA[i]['text']}")
    print(f"  {len(DATA[i]['text'].split())} words, {len(b)} tokens")
    print("  " + " ".join(b[:12]) + " ...")

300 fictional support tickets, simulator vocabulary 177 pieces

Iyi gunler, odememizin durumu gecikmeli ve tekrar yazdim ama sonuc yok. geri donun lutfen. Iyi gunler.
  16 words, 29 tokens
  iyi gun ler , odeme miz in durum u gecikme li ve ...

Merhaba, tarifemizin durumu yuksek ve bu konuda hala bir cevap yok. kayit acin lutfen. Iyi gunler.
  16 words, 26 tokens
  merhaba , tarife miz in durum u yuksek ve bu konu da ...

Iyi gunler, vananin durumu bozuk ve bu konuda hala bir cevap yok. ekip gonderin lutfen. Tesekkur ederim.
  17 words, 27 tokens
  iyi gun ler , vana nin durum u bozuk ve bu konu ...

A sixteen-word ticket comes to 29 tokens. The gap does not come from a single place: punctuation is counted separately, odememizin (of our payment) is split into three pieces, gecikmeli (delayed) into two. Estimating the length of a text by words is, in this unit, counting systematically short.

The Cost of Agglutinative Structure

Turkish is an agglutinative language: successive suffixes can pile onto a root so that a whole sentence is built in a single word. For the reader this is brevity, for the one counting tokens it is cost. Six roots are measured across six inflection steps.

FORMS = {
    "sayac": "sayac sayaclar sayacimiz sayacimizin sayaclarimizin sayaclarimizdaki",
    "kapak": "kapak kapaklar kapagimiz kapaginin kapaklarimizin kapaklarimizdaki",
    "fatura": "fatura faturalar faturamiz faturamizin faturalarimizin faturalarimizdaki",
    "odeme": "odeme odemeler odememiz odememizin odemelerimizin odemelerimizdeki",
    "tarife": "tarife tarifeler tarifemiz tarifemizin tarifelerimizin tarifelerimizdeki",
    "abone": "abone aboneler abonemiz abonemizin abonelerimizin abonelerimizdeki",
}
HEADERS = ("plain", "pl", "poss", "poss+", "pl+", "pl++")
print(f"{'root':<8} " + " ".join(f"{b:>5}" for b in HEADERS))
total = [0] * 6
for k, s in FORMS.items():
    n = [len(tokenize(b)) for b in s.split()]
    total = [a + b for a, b in zip(total, n)]
    print(f"{k:<8} " + " ".join(f"{x:>5}" for x in n))
print(f"{'total':<8} " + " ".join(f"{x:>5}" for x in total))

words = sum(len(x["text"].split()) for x in DATA)
tokens = sum(len(tokenize(x["text"])) for x in DATA)
root_tokens = sum(len([b for b in tokenize(x["text"]) if b not in SUFFIXES]) for x in DATA)
print(f"\n{TICKETS} tickets: {words} words, {tokens} tokens, ratio {tokens / words:.4f}")
print(f"with suffix pieces removed {root_tokens} tokens, ratio {root_tokens / words:.4f}")
print(f"suffix share {(tokens - root_tokens) / tokens:.4f}")

root     plain    pl  poss poss+   pl+  pl++
sayac        1     2     2     3     4     5
kapak        1     2     3     4     4     5
fatura       1     2     2     3     4     5
odeme        1     2     2     3     4     5
tarife       1     2     2     3     4     5
abone        1     2     2     3     4     5
total        6    12    13    19    24    30

300 tickets: 4674 words, 8334 tokens, ratio 1.7831
with suffix pieces removed 6176 tokens, ratio 1.3214
suffix share 0.2589

Six roots take 6 tokens in plain form and 30 tokens fully inflected: the same six concepts grow fivefold. The word count never changed; the only thing that changed is the suffixes. The kapak (cover) row shows one more detail: in the form kapagimiz the consonant softens, so the root no longer matches the vocabulary piece kapak and one extra token is paid.

At the dataset level the ratio is 1.7831: each word comes to 1.78 tokens on average. When suffix pieces are dropped from the sequence, 8,334 tokens fall to 6,176; the suffix share is 0.2589. Writing the same information in a language that is not agglutinative would largely erase this share. This is not a judgment about the language but a budget fact: the same window takes in less of a Turkish text.

When the Same Information Is Shortened

The information a ticket carries is three fields: object, status, request. Courtesy phrases, conjunctions and repeated sentences do not change these three, yet they pay tokens. A scaffolding can be built: drop the pieces whose document frequency exceeds a threshold. A frequent piece does not discriminate; a rare one does.

DOCUMENTS = [tokenize(x["text"]) for x in DATA]
df = {}
for b in DOCUMENTS:
    for t in set(b):
        df[t] = df.get(t, 0) + 1 / TICKETS
STATUS_ROOTS = {tokenize(d)[0] for _, ds in OBJECTS.values() for d in ds}


def read_field(b):
    def first(names):
        return next((t for t in b if t in names), None)
    return (first(set(OBJECT_NAMES)), first(STATUS_ROOTS), first(set(REQUEST_ROOTS)))


def shorten(b, threshold):
    return [t for t in b if df[t] < threshold]


def call(tokens, seed, max_output=40):
    """The simulator generates step by step; it stops when it reaches the stop
    condition or the maximum output length. The output length depends on the seed."""
    r, base, n_out = rng(seed + len(tokens)), 6 + len(tokens) // 3, 0
    while n_out < max_output and (n_out < base or r() >= 0.32):
        n_out += 1
    return {"input": len(tokens), "output": n_out,
            "stop": "limit" if n_out == max_output else "condition"}


def measure(name, lists):
    correct = sum(sum(a == b for a, b in zip(read_field(b), DATA[i]["field"]))
                  for i, b in enumerate(lists))
    n = sum(len(b) for b in lists)
    print(f"{name:<28} {n:>6} {n / TICKETS:>8.2f} {correct / (3 * TICKETS):>8.4f}")


print(f"{'scaffolding':<28} {'tokens':>6} {'ticket':>8} {'field':>8}")
measure("bare call", DOCUMENTS)
measure("drop frequent pieces (0.50)", [shorten(b, 0.50) for b in DOCUMENTS])
measure("drop frequent pieces (0.30)", [shorten(b, 0.30) for b in DOCUMENTS])

k = shorten(DOCUMENTS[2], 0.50)
print("\n" + " ".join(DOCUMENTS[2]))
print(" ".join(k))
for t in (SEED, SEED + 1):
    y = call(k, t)
    print(f"seed {t}  input {y['input']} tokens  output {y['output']} tokens"
          f"  stop {y['stop']}")

scaffolding                  tokens   ticket    field
bare call                      8334    27.78   0.9722
drop frequent pieces (0.50)    3505    11.68   0.9722
drop frequent pieces (0.30)    1868     6.23   0.8611

merhaba , tarife miz in durum u yuksek ve bu konu da hala bir cevap yok . kayit ac in lutfen . iyi gun ler .
merhaba tarife yuksek konu da hala bir cevap kayit ac
seed 20260218  input 10 tokens  output 11 tokens  stop condition
seed 20260219  input 10 tokens  output 10 tokens  stop condition

The triple is read from here. The bare call spends 27.78 tokens per ticket and its field hit rate is 0.9722. The scaffolding drops to 11.68 tokens per ticket at the 0.50 threshold — a gain of 0.5794 — and does not lower the hit rate at all: all sixteen dropped pieces are courtesy phrases, conjunctions or inflection suffixes. The cost is a preprocessing step, a frequency table and a loss of readability; because the number does not fall, this scaffolding pays off.

When the threshold is pulled to 0.30 the picture turns. Tokens fall to 6.23, the gain becomes 0.7759, but the hit rate drops to 0.8611: the document frequency of the yuksek (high) status is 0.3333, and because it stays above the threshold it is dropped. The most frequent status is also the most asked-about status. Adding scaffolding does not always improve things, and whether it improves them is seen only when it is written together with its cost.

The bare call staying at 0.9722 is also a finding. All 25 missed fields belong to the kapak (cover) object: because the root softens in the form kapaginin (of its cover), the kapak piece never appears in the token sequence. Agglutinative structure does not only inflate the count; it can also erase the root from the token stream.

The last two lines separate the uncertainty. When the same prompt is called with two different seeds, the input token count is 10 in both — tokenization is a procedure, independent of the run. The output is 11 and 10, and both ended with the stop condition. Of the two quantities countable on the model’s surface, one is independent of the run and the other is not; the budget calculation is built on this distinction.

Summary

• A token is the input unit of text for the model and the unit of cost of this curriculum. The simulator applies leftmost longest match with a vocabulary of 177 pieces; the tickets are fictional, seed 20260218.
• The same 300 tickets come to 4,674 words and 8,334 tokens: 1.7831 per word. An estimate made from the word count systematically counts short in this unit.
• Agglutinative structure is a measurable cost: six roots come to 6 tokens in plain form and 30 tokens fully inflected, and at the dataset level the suffix share is 0.2589.
• The bare call gives 27.78 tokens per ticket and a field hit rate of 0.9722. The scaffolding that drops frequent pieces at the 0.50 threshold falls to 11.68 tokens and keeps the hit rate; the 0.30 threshold falls to 6.23 tokens but lowers the hit rate to 0.8611.
• The input token count is independent of the run, the output length is not: the same prompt gives 10 input tokens and 11 and 10 output tokens on two seeds.

Next Step

The text was split into tokens and the cost of every token was counted. There is still only a sequence in the middle: what the model does with this sequence, how a token is related to another one a hundred tokens earlier was not asked. The next lesson covers conceptually why the attention mechanism can use distant context and counts its cost: how the number of comparisons grows as the context lengthens, and at what distance a cheap procedure content with a fixed window collapses.