Unit 02 of 02
How Language Models Work
Measuring the surface a given language model exposes to the outside: the number of tokens the same Turkish text costs and how its agglutinative structure enlarges it, the cost of attention being able to use distant context as a number of comparisons, how many examples it takes for a task set to improve in pretraining and in fine-tuning, what the most likely choice and sampled choice give in two runs on the same prompt, what dropping from the head, dropping from the middle and summarizing gain in tokens and lose in hit rate when the context window is exceeded, the collapse of the bare call on counting and exact arithmetic tasks, the fabrication rate of claims not based on a document in an ungrounded response and the cost of three mitigations, the comparison of open- and closed-weight access on three metrics, how model selection changes as the metric weight changes, and the token and hit rate comparison of four scaffoldings.
Published Updated
Lessons
- 01 Tokens Measuring the input unit of text for the model as a unit of cost: 300 fictional support tickets come to 4,674 words and 8,334 tokens, 1.7831 tokens per word; with suffix pieces removed 6,176 tokens remain and the suffix share comes out at 0.2589. Six roots take 6 tokens in plain form and 30 tokens as plural and possessive suffixes pile on. A bare call gives 27.78 tokens per ticket and a field hit rate of 0.9722, while the scaffolding that drops frequent pieces keeps the same hit rate with 11.68 tokens at the 0.50 threshold and falls to 6.23 tokens at 0.30, but the hit rate drops to 0.8611. 13 min read
- 02 Transformer Intuition Measuring why attention can use distant context and what it costs as a number of comparisons: in 240 fictional ticket streams the topic object is on average 63.9 tokens from the end, and the reader that looks at the last 16 tokens gives a hit rate of 0.0000, returning the wrong object in 83 streams and nothing in 157. The last 64 tokens rise to 0.4917 but stay at 0.0000 beyond 65 tokens; the weight that decays with distance stays at 0.3333 although it sees every position. The weight independent of distance gives 1.0000 and its cost is 70.9 comparisons: in a full pass, when the context doubles the comparisons quadruple, and at 1024 tokens it is 8.26 times the fixed window. 7 min read
- 03 Pretraining and Fine-Tuning Counting a task set split in two and how many examples each half needs to improve: the simulator pretrained on a fictional general-text corpus of 1,600 sentences, without seeing a single domain example, gives a next-token hit rate of 0.3186 on general text and 0.1983 on domain text; the same measure without pretraining is 0.1296. Fine-tuning with four hundred domain examples raises domain text to 0.5437 but drops general text to 0.2692. The organization's own fault-code mapping is never solved by pretraining: the hit rate stays at 0.0900, below the 0.1200 share of the most frequent code, and the 0.90 threshold is passed only with 400 labeled examples. 13 min read
- 04 Inference The cost of producing a response step by step and a count for the selection rule: in 200 fictional tickets greedy decoding takes 6.43 tokens and 211.1 comparisons per response with a hit rate of 1.0000. When the maximum output length is pulled down to 4, all responses hit the limit and the hit rate drops to 0.0000, to 0.5450 at 6, to 0.8100 at 7. On the same prompt, as the distribution temperature of sampling rises from 0.2 to 2.0 the hit rate falls from 0.9900 to 0.5600 and the share of responses that change between two runs rises from 0.5550 to 0.8600; with greedy decoding this share is 0.0000. With the input fixed at 30 tokens, when the output grows to 512 tokens the output itself accounts for 0.8953 of the comparisons. 4 min read
- 05 Context Window Measuring what happens when the window is exceeded and how many tokens three truncation decisions save and how much they lower the hit rate: in a window of 256 tokens, 0.5967 of 300 fictional prompts do not fit, and the bare call gives a full-response rate of 0.3833 at 282.3 tokens. Dropping from the head gives 0.7433 at 217.5 tokens, dropping from the middle gives 0.7367 at the same tokens, and summarizing raises the code hit rate to 0.9667 at 100.6 tokens but drops the step hit rate to 0.0000. Which decision truncation makes depends on the position of the evidence: dropping from the head gives 0.5351 in the head bucket, dropping from the middle gives 0.3627 in the middle bucket. When the window goes from 128 to 384, the full response rises from 0.3767 to 0.9867. 13 min read
- 06 Capabilities and Limits Measuring five task classes with the bare call on twenty-four fictional support requests: summarization 0.7500, classification 0.8750, ranking 0.6250, counting 0.4167 and exact arithmetic 0.2917. When the seed is changed the number of the first three classes moves, the last two do not move at all, and the number of requests whose response changed comes out 0/24. The counting hit rate equals exactly the 10/24 requests that carry no inflected form, the exact-arithmetic hit rate equals exactly the 7/24 requests that carry no carry across a piece boundary. The three-run majority scaffolding changes nothing in these two classes with three times the tokens, the procedure scaffolding that takes the calculation out of the tokens carries both to 1.0000. 5 min read
- 07 Hallucination When twenty-four questions are asked over a fictional document pool of twenty-seven facts, the ungrounded response gets a hit rate of 0.2917 and a fabrication rate of 0.6667, and the response changes in 16/24 requests across two runs. Three mitigations are measured side by side: grounding raises the hit rate by +0.3750 and brings fabrication down to 0.3333, claim verification brings fabrication down to 0.0857 but its cost is two calls and 66.2 tokens, and the abstention option drops the hit rate 0.0417 below the baseline and leaves 10/24 requests unanswered. All of the unsupported claims left after grounding come from questions that have no counterpart in the document, and the verification scaffolding gives a difference of 0/24 across two runs. 7 min read
- 08 Closed- and Open-Weight Models Two access modes are compared on three criteria with numbers, and the criteria do not converge on a single form. The control held is 3/9 settings in the closed-weight form and 9/9 in the open-weight form. On unit cost the closed form holds a hit rate of 0.7917 at 41.8 tokens per request, while the open form reaches the same hit rate only after six examples are added, at 179.8 tokens, and the ratio comes out at 4.30. On the data boundary the closed form sends 24/24 requests, 1004 tokens and 72 numeric fields across the organization boundary, and the open form sends none. Two criteria pick the open-weight form, one picks the closed-weight form, and the three numbers have no common unit. 5 min read
- 09 Model Selection Three candidates are built at the simulator level and counted on four criteria: small capability 0.5000 hit rate / 89.5 equivalent tokens / 7.29 steps / 0 requests crossing, medium capability 0.5833 / 195.3 / 7.00 / 0, large capability 0.7917 / 670.0 / 14.00 / 24. Across five weight sets the winner is medium three times, large once and small once. The weight sweep gives the thresholds: when the hit-rate weight reaches 4 the choice moves from medium to large, when the cost weight reaches 2 it moves from medium to small, and the latency and data-boundary weights do not change the choice up to twelve. When the same computation is repeated with seed 20260219, the winner of the hit-rate weighted set turns from large back to medium. 5 min read
- 10 Token-Based Cost Over a fictional stream of ninety-six requests, four scaffoldings are compared in tokens and hit rate: the long prompt gives 0.7083 hit rate at 159.8 tokens per request, the short prompt with examples gives the same hit rate at 103.8 tokens, summarize and send gets down to 74.0 tokens with 0.8021, the hard summary drops the hit rate to 0.0833 at 68.0 tokens, and the cache gives 0.7500 at 17.7 tokens. The total tokens fall from 15,341 to 1,702 and the model calls fall from 96 to 23. When the budget drops to 7,000 tokens only two of the five scaffoldings fit, and the design is not chosen but imposed. The hit rate of the cache comes out at 0.7500 and 0.9583 across the two runs. 5 min read