--- title: 'Introduction to AI Engineering' source: 'https://academia.sh/en/courses/introduction-to-ai-engineering' language: en updated: '2026-10-21T05:02:30+00:00' license: 'CC BY-SA 4.0' lessons: 14 outcomes: - 'Distinguishes AI engineering from machine learning engineering by the number of settings in hand' - 'Explains how a language model works along the axes of tokens, context window and inference' - 'Ties model choice to the criteria of capability, latency, token cost and data limits' --- # Introduction to AI Engineering Measuring a component that cannot be looked inside from its surface: the difference and cost between a bare call and a small scaffolding on the same task, the number of settings in the hands of the training and the using approach, the output that changes between two runs of the same input and three design decisions that narrow the uncertainty, separating a measurable capability claim from an unmeasurable general claim, the number of tokens a Turkish text costs, the cost of attention using distant context, which problem is solved in pretraining and which in fine-tuning, two runs of greedy decoding and sampling, what three truncation decisions gain and drop when the context window is exceeded, where the call without scaffolding collapses by task class, the fabrication rate and the cost of three mitigations, comparing two access modes metric by metric, the model choice that changes as the weights change, and the design that the budget imposes. ## Field and Roles Measuring with numbers what it means to build a product without training a model and to design around uncertain output: the difference and cost of a bare call versus a small scaffolding on the same support ticket task, the number of settings and the boundary of responsibility in the hands of a training approach and a using approach, in how many requests the output changes between two runs of the same input and how much of that the user sees, how much offering options, imposing a format, and grounding narrow the uncertainty, the separation of a measurable capability claim from an unmeasurable general claim, and how many of them can be tied to a test. ### Who Is an AI Engineer Source: https://academia.sh/en/courses/introduction-to-ai-engineering/who-is-an-ai-engineer Establishing the unit of measurement for building products without training a model: a language model simulator is written and the surface of a given component is read with five numbers. On the same 120 fictional support tickets, a bare call gives 0.5167 label hit rate and 0.3583 unlabeled response share, while a small scaffolding with a label set and a format constraint gives 0.7750 and 0.0000. The cost is written side by side: input tokens rise from 2666 to 6222, output tokens fall from 1648 to 218, the call count rises from 120 to 124, total tokens rise 1.49 times, and the cost per correct label stays near 69 tokens. Over 121 lessons, the previous curriculum built a model, measured it, explained it, and counted what it did to whom. Its closing left one gap written down: all of that happened **in a single run, on a single machine, inside a notebook**. A model turning into a **system** — getting versioned, gaining an interface that others call, being rolled back when it breaks, having its cost counted — was never covered. This curriculum opens from there and adds one more thing: the model is **no longer something you trained** either. It is a given component whose weights you cannot see, whose settings you cannot change, and whose insides you cannot inspect. The only thing left in your hands is its **surroundings**: what you say, what you put next to it, how you read its output, when you give up. What this curriculum measures is not the model but those surroundings; they will be called the **scaffolding**. This lesson sets up the unit of measurement and writes the **language model simulator** that will be used across seven courses. - **FR1.** No lesson in this curriculum calls a real model endpoint. The code calls a **simulator** written with the standard library. The simulator does not imitate a product; it carries the **measurable surface** of an endpoint. - **FR2.** The simulator's tokenizer is fictional: words are split on whitespace and pieces longer than five characters are cut. Tokenization itself was measured in the Natural Language Processing and Computer Vision course and is not repeated here; here the token is only a **unit of cost**. - **FR3.** The surface is five measures: input tokens, output tokens, the **context window** (240 tokens), maximum output length (24 tokens), and status. - **FR4.** The simulator is determined by a seed (**20260218**). The same prompt on the same seed gives the same response; the numbers do not drift from run to run. **Uncertainty is measured by changing the seed.** - **FR5.** Two failures are built in: the rate limit depends on the call count (every thirty-seventh call), and the timeout depends on prompt length. Both enter the failure share. - **FR6.** The support tickets are **fictional**: 120 records, five topics, and a misleading sentence appended to a portion of the records. The regional measurement network of the earlier curricula (Data Analytics and Machine Learning and Data Science) turns into a support assistant here. - **FR7.** The simulator's internal trace table holds a few **misleading links**. They are the fictional counterpart of the concept pairs a model systematically confuses, and they cannot be removed from the outside. - **FR8.** The criterion is the **label hit rate**. If a label cannot be read from the response, the record counts as wrong and is additionally shown in the **unlabeled response share**. #### The Measurable Surface of a Model A given component cannot be measured from the inside. Its weights are invisible, its training data is unknown, and the reasoning behind its decision cannot be read. One place to measure remains: the **surface the component presents to the outside**. How many tokens went in, how many came out, where the limit was, what the same input produced in two runs, which status the call returned with. **This is the rule of the course: a given component is measured from its surface.** The first block sets up this surface. What gets called is a language model simulator; no request goes to a real endpoint. ```python # SIMULATOR. This code does not call a real model endpoint; it is a language model # simulator written with the standard library. The support tickets are also FICTIONAL. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUTPUT = 240, 24 # context window and maximum output, in tokens def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): # FICTIONAL tokenizer: pieces of at most five characters tok = [] for s in text.lower().replace(",", " , ").split(): while len(s) > 5: tok.append(s[:5]) s = s[5:] tok.append(s) return tok def trace_id(text): # trace that depends on the prompt content, not on the run h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["invoice", "fault", "meter", "tariff", "outage"] TEMPLATES = { "invoice": ["the invoice amount this month is very high compared to last month", "i paid my invoice but the system shows a debt", "where can i see the invoice details"], "fault": ["the water is not flowing at home and there is a fault i think", "the connection is broken and it is not working at all", "i opened a fault ticket but the team never came"], "meter": ["the meter reading was entered wrong and the index does not match", "i want an appointment for the meter change", "the index on the meter and the invoice are not the same"], "tariff": ["i want to learn which tariff tier i am on", "what do i need to do for a tariff change", "does the unit price change by subscriber type"], "outage": ["there is an outage in the neighborhood when will it be back", "where is the planned outage announcement tracked", "we have been having outages since the evening"], } EXTRAS = ["the same issue happened last month too", "i am not satisfied with this", "i urgently need information", "i do not have my record number"] TICKETS = [] for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = TEMPLATES[k][int(r() * 3)] if r() < 0.34: m += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": m, "topic": k}) CONCEPTS = TOPICS + ["payment", "complaint", "information", "general"] TRACES = {"invoi": "invoice", "amoun": "invoice", "detai": "invoice", "fault": "fault", "broke": "fault", "worki": "fault", "flowi": "fault", "meter": "meter", "index": "meter", "readi": "meter", "tarif": "tariff", "unit": "tariff", "subsc": "tariff", "tier": "tariff", "outag": "outage", "annou": "outage", "neigh": "outage", "debt": "payment", "paid": "payment", "price": "payment", "satis": "complaint", "issue": "complaint", "urgen": "complaint", "infor": "information", "learn": "information", "where": "information"} TRAPS = {"chang": "tariff", "recor": "fault", "high": "tariff", "shows": "invoice", "team": "outage", "track": "tariff", "enter": "invoice"} FREE_FORM = ["your request seems to relate to {k}", "this falls under {k} , forwarded to the relevant team", "your record has been received , we will review the matter and get back to you"] COUNTER = [0] def call(prompt, seed=SEED, temperature=0.6, max_output=MAX_OUTPUT): COUNTER[0] += 1 tok = tokenize(prompt) if len(tok) > WINDOW: return {"status": "context_overflow", "input": len(tok), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: return {"status": "rate_limit", "input": len(tok), "output": 0, "response": ""} r = rng(seed ^ trace_id(prompt)) if len(tok) > 90 and r() < 0.04: return {"status": "timeout", "input": len(tok), "output": 0, "response": ""} cand = [k for k in CONCEPTS if k[:5] in tok] # concepts named in the prompt if len(cand) < 3: cand = CONCEPTS p = {k: 0.15 for k in cand} for t in tok: for table, w in ((TRACES, 1.0), (TRAPS, 1.3)): if table.get(t) in p: p[table[t]] += w if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) # most likely choice else: e = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [v / sum(e) for v in e])] y = k if "only" in tok else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k) out = tokenize(y) if len(out) > max_output: # stop condition out = out[:max_output] y = " ".join(out) return {"status": "ok", "input": len(tok), "output": len(out), "response": y} def read_label(y): # extract a label from the response for s in y.lower().split(): if s in TOPICS: return s return None first = TICKETS[0] pieces = tokenize(first["text"]) print(f"FICTIONAL ticket set: {len(TICKETS)} records, {len(TOPICS)} topics") print(f"example ticket {first['id']}: {first['text']}") print(f"token count {len(pieces)}, first six pieces {pieces[:6]}") sample = call("Answer this support ticket: " + first["text"]) print(f"\nstatus {sample['status']}, input {sample['input']} tokens, output {sample['output']} tokens") print(f"response: {sample['response']}") prompt1 = "Answer this support ticket: " + TICKETS[3]["text"] for sd in (SEED, SEED + 1): print(f"seed {sd}: {call(prompt1, seed=sd)['response']}") short = call("Answer this support ticket: " + first["text"], max_output=4) print(f"\nmaximum output 4 tokens: {short['response']}") long_call = call(" ".join(x["text"] for x in TICKETS[:20])) print(f"prompt over the window: status {long_call['status']}, input {long_call['input']} (limit {WINDOW})") ``` ``` FICTIONAL ticket set: 120 records, 5 topics example ticket 4001: where is the planned outage announcement tracked token count 12, first six pieces ['where', 'is', 'the', 'plann', 'ed', 'outag'] status ok, input 19 tokens, output 10 tokens response: your request seems to relate to outage seed 20260218: your request seems to relate to outage seed 20260219: this falls under fault , forwarded to the relevant team maximum output 4 tokens: your reque st seems prompt over the window: status context_overflow, input 321 (limit 240) ``` Each of the seven lines after the header shows a limit. A ticket of seven words comes to **twelve tokens**; a word longer than five characters is split into pieces. The number of words and the number of tokens are not the same thing, and the cost is read from the second. The fourth and fifth lines are the most uncomfortable feature of this curriculum. **The same prompt, on two seeds, gave two different responses**: the first names `outage`, a label that is not the ticket's topic, while the second says `fault`. The ticket's real topic is `fault`. So the correct response came out once and did not come out once, and in between the prompt never changed. A component giving two different responses to the same input is the justification for everything that will be built around it. The last two lines are two hard limits. When the maximum output length is lowered to four, the response is cut after its fourth token: **the stop condition cut the response in the middle** and the output stopped on a fragment, not a finished answer. When twenty tickets are joined and sent, 321 tokens exceed the context window and the call returns without producing any response. Neither is the model "thinking wrong"; both are numbers given by the surface. #### The Difference Between a Bare Call and Scaffolding The task is fixed: each of 120 fictional support tickets will be assigned to one of five topics. Two paths are put side by side. The **bare call** is the plainest use — sending the ticket as it is and reading the response. The **scaffolding** is three small parts: stating the label set in the prompt, imposing the format, and retrying once on an unreadable response. ```python INSTRUCTION = ("Assign the ticket to one of these labels: invoice , fault , meter , tariff , " "outage . Write only the label as the response . Ticket: ") def measure(scaffolding): COUNTER[0] = 0 tin = tout = hits = empty = retries = failures = 0 for x in TICKETS: prompt = (INSTRUCTION if scaffolding else "Answer this support ticket: ") + x["text"] s = call(prompt) tin, tout = tin + s["input"], tout + s["output"] failures += s["status"] != "ok" e = read_label(s["response"]) if e is None and scaffolding: # format could not be read: retry once retries += 1 s = call(prompt, seed=SEED + 1) tin, tout = tin + s["input"], tout + s["output"] failures += s["status"] != "ok" e = read_label(s["response"]) empty += e is None hits += e == x["topic"] return {"hit": hits / len(TICKETS), "empty": empty / len(TICKETS), "calls": COUNTER[0], "input": tin, "output": tout, "retries": retries, "failure": failures / COUNTER[0]} A, B = measure(False), measure(True) print(f"{'measure':<24} {'bare call':>13} {'scaffolding':>11} {'difference':>10}") for name, key, fmt in (("label hit rate", "hit", "{:.4f}"), ("unlabeled response share", "empty", "{:.4f}"), ("call count", "calls", "{:d}"), ("input tokens", "input", "{:d}"), ("output tokens", "output", "{:d}"), ("failure share", "failure", "{:.4f}")): f = B[key] - A[key] print(f"{name:<24} {fmt.format(A[key]):>13} {fmt.format(B[key]):>11} " f"{('{:+.4f}' if '.' in fmt else '{:+d}').format(f):>10}") tA, tB = A["input"] + A["output"], B["input"] + B["output"] print(f"\ntotal tokens {tA} -> {tB} ({tB / tA:.2f}x), retries {B['retries']}") print(f"cost per correct label: {tA / A['hit'] / len(TICKETS):.1f} -> " f"{tB / B['hit'] / len(TICKETS):.1f} tokens") ``` ``` measure bare call scaffolding difference label hit rate 0.5167 0.7750 +0.2583 unlabeled response share 0.3583 0.0000 -0.3583 call count 120 124 +4 input tokens 2666 6222 +3556 output tokens 1648 218 -1430 failure share 0.0250 0.0242 -0.0008 total tokens 4314 -> 6440 (1.49x), retries 4 cost per correct label: 69.6 -> 69.2 tokens ``` The first row is the scaffolding's gain: **from 0.5167 to 0.7750**, an increase of **0.2583**. The second row says where the gain comes from. In the bare call, **0.3583 of the responses** carried no readable label at all; either the response closed on a word outside the five labels, or, in the form of "your record has been received , we will review the matter and get back to you", it carried no topic name. When the label set is written into the prompt, this share drops to **zero**. The gain does not come from the model thinking better but from **the answer becoming readable**. #### A Scaffolding Without a Written Cost Counts as Unmeasured The lower half of the same table is the cost, and in this curriculum it is as binding as the upper half. Input tokens went **from 2666 to 6222**: adding the instruction to the start of every ticket more than doubled the prompt length. Output tokens went the other way, **from 1648 to 218**; producing a one-word response is several times cheaper than producing a sentence. The call count went **from 120 to 124**; all four extra calls are requests that hit the rate limit and were retried. The failure share stayed roughly the same on both paths, because the rate limit is a property of the **surface**, not of the scaffolding. The last row is the real number of this lesson. Total tokens rose **1.49 times**: the scaffolding spends half again as many tokens on the same 120 tickets. The hit rate rose by almost the same factor, so the cost per correct label barely moves: **from 69.6 tokens to 69.2 tokens**. The scaffolding did not pay for its accuracy with a cheaper label; it **bought more correct labels at a higher total bill**. A report that writes only the hit rate, or only the per-label cost, hides the 1.49 times; a report that writes only one of the two numbers is a defense, not a measurement. **This is the rule of the curriculum: the number of an artificial intelligence system is not the model's number but the scaffolding's number; a scaffolding without a written cost counts as unmeasured.** The work of the AI engineer is visible in this table. The model is given, and in this table it never changed: the same simulator, the same seed, the same weights. The only thing that changed is its **surroundings** — the label set written into the prompt, the imposed format, the retry decision made on an unreadable response, and the latency class held to a single call. The settings learned over 121 lessons are not in hand here; other settings take their place, and all of them sit **outside** the model. The next lesson puts these two sets of settings side by side and counts them. #### Summary - A given component cannot be measured from the inside; it is measured from its **surface**: input tokens, output tokens, context window, stop condition, and status. The code of this curriculum calls a **simulator**. - The model is an uncertain component. The same prompt gave two different responses on two seeds; in one the correct label came out, in the other a wrong label. - On the same 120 fictional tickets, the bare call gives a label hit rate of **0.5167**, and **0.3583** of the responses have no readable label. The scaffolding made of a label set, a format constraint, and a single retry gives **0.7750** and **0.0000**. - The cost is written side by side: input tokens go from 2666 to 6222, calls from 120 to 124, output tokens fall from 1648 to 218, and total tokens rise **1.49 times**. - The cost per correct label stays nearly flat, **from 69.6 to 69.2 tokens**, while the total bill rises 1.49 times. A scaffolding without a written cost counts as unmeasured. #### Next Step In this lesson the model never changed and all of the gain came from outside it. This is the exact opposite of the previous curriculum: there the gain came from inside the model and the outside was held fixed. Both approaches can solve the same problem, but what they hold in their hands is not the same. The next lesson counts it: on the same support ticket problem, how many settings the training approach has, how many settings the using approach has, how many of those settings really move the number, and where **who holds which decision** separates in the two approaches. ### The Difference Between AI Engineering and Machine Learning Engineering Source: https://academia.sh/en/courses/introduction-to-ai-engineering/ai-and-machine-learning-engineering-difference Measuring the boundary of responsibility by the number of settings: on the same support ticket problem, two of the training approach's four settings and three of the using approach's four settings (four of five rows once the seed is counted) move the number by more than 0.0200. The training approach asks for 48 labeled examples and reaches 1.0000, while the using approach gives 0.7778 with zero labeled examples. No item of the two setting lists is shared. Retrying moves the number by 0.0139 and adds two calls and 111 tokens, while the seed, which is out of one's hands, moves the number by 0.0417. The previous lesson raised the number from 0.5167 to 0.7750 without changing the model at all. All of the gain came from outside the model. In the previous curriculum the opposite held: the gain came from inside the model and the outside was held fixed. Both approaches can solve the same support ticket problem, but what they hold in their hands is not the same. This lesson ties that difference to a number. Both approaches are run on the same 72-record test set, and the **setting in each one's hands** is turned one by one. What is measured is not which is better but **who holds which decision**: how many settings there are, how many really move the number, and how much what is out of one's hands moves it. - **FR9.** The fiction and the simulator carry over unchanged from the previous lesson: 120 fictional support tickets, five topics, seed **20260218**. The training set is the first 48 records and the test set is the remaining **72 records**. Both approaches are read on the same test set. - **FR10.** The training approach's model is a **count table**: it keeps how many times each token in the training set appears under which topic and names the topic with the highest total evidence. It is a learning procedure and its weights are determined by training. - **FR11.** The Supervised Learning course measured tuning variables (depth, minimum leaf, class weight, tuning budget). That procedure is **not repeated** here; here only **who holds the setting** is counted. - **FR12.** The using approach calls the simulator and never touches the inside of the model. It builds no count table and sees no labeled examples. - **FR13.** A setting "moves the number" when the absolute value of the hit rate difference between its two ends is **greater than 0.0200**. The threshold is a decision and is held fixed. - **FR14.** The seed is not a setting; it is **the measure of the simulator's uncertainty**. It is placed in the table to show how much what is out of one's hands moves. - **FR15.** The cost columns exist only for the using approach, because the cost of the training approach is not calls and tokens but **labeled examples**. The two cost units are not converted into each other. #### The Settings in the Training Approach's Hands In the training approach the model is an object you produce. The settings in your hands look inside it: which data it is built from, which unit it counts with, which evidence it ignores, how it smooths sparse observations. The block below sets up the simulator and the fiction, then writes the learning procedure and turns four settings at two ends. ```python # SIMULATOR. This code does not call a real model endpoint; it is a language model # simulator written with the standard library. The support tickets are also FICTIONAL. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUTPUT = 240, 24 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): tok = [] for s in text.lower().replace(",", " , ").split(): while len(s) > 5: tok.append(s[:5]) s = s[5:] tok.append(s) return tok def trace_id(text): h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["invoice", "fault", "meter", "tariff", "outage"] TEMPLATES = { "invoice": ["the invoice amount this month is very high compared to last month", "i paid my invoice but the system shows a debt", "where can i see the invoice details"], "fault": ["the water is not flowing at home and there is a fault i think", "the connection is broken and it is not working at all", "i opened a fault ticket but the team never came"], "meter": ["the meter reading was entered wrong and the index does not match", "i want an appointment for the meter change", "the index on the meter and the invoice are not the same"], "tariff": ["i want to learn which tariff tier i am on", "what do i need to do for a tariff change", "does the unit price change by subscriber type"], "outage": ["there is an outage in the neighborhood when will it be back", "where is the planned outage announcement tracked", "we have been having outages since the evening"], } EXTRAS = ["the same issue happened last month too", "i am not satisfied with this", "i urgently need information", "i do not have my record number"] TICKETS = [] for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = TEMPLATES[k][int(r() * 3)] if r() < 0.34: m += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": m, "topic": k}) CONCEPTS = TOPICS + ["payment", "complaint", "information", "general"] TRACES = {"invoi": "invoice", "amoun": "invoice", "detai": "invoice", "fault": "fault", "broke": "fault", "worki": "fault", "flowi": "fault", "meter": "meter", "index": "meter", "readi": "meter", "tarif": "tariff", "unit": "tariff", "subsc": "tariff", "tier": "tariff", "outag": "outage", "annou": "outage", "neigh": "outage", "debt": "payment", "paid": "payment", "price": "payment", "satis": "complaint", "issue": "complaint", "urgen": "complaint", "infor": "information", "learn": "information", "where": "information"} TRAPS = {"chang": "tariff", "recor": "fault", "high": "tariff", "shows": "invoice", "team": "outage", "track": "tariff", "enter": "invoice"} FREE_FORM = ["your request seems to relate to {k}", "this falls under {k} , forwarded to the relevant team", "your record has been received , we will review the matter and get back to you"] COUNTER = [0] def call(prompt, seed=SEED, temperature=0.6, max_output=MAX_OUTPUT): COUNTER[0] += 1 tok = tokenize(prompt) if len(tok) > WINDOW: return {"status": "context_overflow", "input": len(tok), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: return {"status": "rate_limit", "input": len(tok), "output": 0, "response": ""} r = rng(seed ^ trace_id(prompt)) if len(tok) > 90 and r() < 0.04: return {"status": "timeout", "input": len(tok), "output": 0, "response": ""} cand = [k for k in CONCEPTS if k[:5] in tok] if len(cand) < 3: cand = CONCEPTS p = {k: 0.15 for k in cand} for t in tok: for table, w in ((TRACES, 1.0), (TRAPS, 1.3)): if table.get(t) in p: p[table[t]] += w if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) else: e = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [v / sum(e) for v in e])] y = k if "only" in tok else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k) out = tokenize(y) if len(out) > max_output: out = out[:max_output] y = " ".join(out) return {"status": "ok", "input": len(tok), "output": len(out), "response": y} def read_label(y): for s in y.lower().split(): if s in TOPICS: return s return None TEST = TICKETS[48:] # test set: 72 records def split_text(text, pieces): return tokenize(text) if pieces else text.lower().split() def train_model(dataset, pieces=True, min_freq=1, smoothing=0.5): # LEARNING procedure: count table counts, freq = {}, {} for x in dataset: freq[x["topic"]] = freq.get(x["topic"], 0) + 1 for t in split_text(x["text"], pieces): c = counts.setdefault(t, {}) c[x["topic"]] = c.get(x["topic"], 0) + 1 majority = max(sorted(freq), key=lambda k: freq[k]) def predict(text): p = {k: 0.0 for k in TOPICS} for t in split_text(text, pieces): c = counts.get(t, {}) if sum(c.values()) < min_freq: continue for k in TOPICS: p[k] += math.log((c.get(k, 0) + smoothing) / (freq.get(k, 1) + 5 * smoothing)) return max(sorted(p), key=lambda k: p[k]) if any(p.values()) else majority return predict def hit_rate_train(n=48, **kw): f = train_model(TICKETS[:n], **kw) return sum(f(x["text"]) == x["topic"] for x in TEST) / len(TEST) print(f"{'TRAINING approach setting':<31} {'A':>7} {'B':>7} {'move':>8}") moves_train = [] for name, a, b in (("training examples 8 / 48", {"n": 8}, {"n": 48}), ("token unit word / piece", {"pieces": False}, {"pieces": True}), ("min frequency 3 / 1", {"min_freq": 3}, {"min_freq": 1}), ("smoothing 0.05 / 0.50", {"smoothing": 0.05}, {"smoothing": 0.5})): va, vb = hit_rate_train(**a), hit_rate_train(**b) moves_train.append(abs(vb - va)) print(f"{name:<31} {va:>7.4f} {vb:>7.4f} {vb - va:>+8.4f}") print(f"settings that move it by more than 0.0200: {sum(v > 0.02 for v in moves_train)}/{len(moves_train)}, " f"labeled examples needed 48, what changes: the count table itself") ``` ``` TRAINING approach setting A B move training examples 8 / 48 0.6250 1.0000 +0.3750 token unit word / piece 0.9861 1.0000 +0.0139 min frequency 3 / 1 0.8611 1.0000 +0.1389 smoothing 0.05 / 0.50 0.9861 1.0000 +0.0139 settings that move it by more than 0.0200: 2/4, labeled examples needed 48, what changes: the count table itself ``` Two of the four settings move the number above the threshold, and the largest move comes from the **number of labeled examples**: 0.6250 with eight examples, **1.0000** with forty-eight. This is both the strength and the bill of the training approach. The most effective way to raise the number is more labeled examples, and labeled examples are something that has to be produced. The remaining three settings also look **inside** the model: which unit it counts with, which evidence is thrown away, how sparse observations are smoothed. All of them are within reach and all of them change the model itself. #### The Settings in the Using Approach's Hands In the using approach no setting looks inside the model. There is no count table, no weights, no training set. What remains is the **surroundings** of the call. ```python INSTRUCTION = ("Assign the ticket to one of these labels: invoice , fault , meter , tariff , " "outage . Write only the label as the response . Ticket: ") def measure_use(instruction=INSTRUCTION, temperature=0.6, seed=SEED, retries=0): COUNTER[0] = 0 hits = tokens = 0 for x in TEST: e = None for j in range(retries + 1): s = call(instruction + x["text"], seed=seed + j, temperature=temperature) tokens += s["input"] + s["output"] e = read_label(s["response"]) if e is not None: break hits += e == x["topic"] return hits / len(TEST), COUNTER[0], tokens print(f"{'USING approach setting':<31} {'A':>7} {'B':>7} {'move':>8} " f"{'calls':>6} {'tokens':>9}") moves_use = [] for name, a, b in (("label set no / yes", {"instruction": "Answer this support ticket: "}, {}), ("format constraint no / yes", {"instruction": INSTRUCTION.replace("only the label", "an answer")}, {}), ("temperature 0.6 / 0.0", {"temperature": 0.6}, {"temperature": 0.0}), ("retries 0 / 1", {"retries": 0}, {"retries": 1}), ("seed 20260218 / 20260219", {"seed": SEED}, {"seed": SEED + 1})): (va, ca, ba), (vb, cb, bb) = measure_use(**a), measure_use(**b) moves_use.append(abs(vb - va)) print(f"{name:<31} {va:>7.4f} {vb:>7.4f} {vb - va:>+8.4f} {cb - ca:>+6d} {bb - ba:>+9d}") print(f"settings that move it by more than 0.0200: {sum(v > 0.02 for v in moves_use)}/{len(moves_use)}, " f"labeled examples needed 0, what changes: everything outside the model") ``` ``` USING approach setting A B move calls tokens label set no / yes 0.5417 0.7778 +0.2361 +0 +1143 format constraint no / yes 0.5000 0.7778 +0.2778 +0 -917 temperature 0.6 / 0.0 0.7778 0.9028 +0.1250 +0 -7 retries 0 / 1 0.7778 0.7917 +0.0139 +2 +111 seed 20260218 / 20260219 0.7778 0.8194 +0.0417 +0 -14 settings that move it by more than 0.0200: 4/5, labeled examples needed 0, what changes: everything outside the model ``` The largest move comes from the **format constraint**: from 0.5000 to 0.7778, an increase of **0.2778**. This row also has by far the largest **minus** sign in the cost column — it saves 917 tokens, while the temperature and seed rows shift the cost by only 7 and 14 tokens. Telling the model to write "only the label" both raises the hit rate and lowers the cost, because a short response is cheap. Writing the label set into the prompt also gains 0.2361, but its cost is **1143 tokens**. The fourth row pays off the second claim of this curriculum for the first time. **Retrying moves the number by 0.0139**, one record out of 72 and below the threshold, and in return asks for two extra calls and 111 tokens. In the plain scaffolding there was already no unreadable response left except the requests that failed at the surface; retrying could only rescue those, and it rescued one of the two. **Adding scaffolding does not always improve enough to matter, but its cost is always certain.** The fifth row is not a setting. The seed is not your setting; it is the simulator's uncertainty, and in this table it **moves the number by 0.0417**. So something that is not in your hands moves the number more than one of the four settings that are. Part of the effort spent on raising a product's number is spent without knowing that the measurement itself moves by this much. #### Who Holds Which Decision **No item of the two lists is shared.** The table below writes this separation at the level of decisions. | Decision | Training approach | Using approach | |---|---|---| | Collecting labeled examples | mandatory, the setting that moves the number most | not needed, works with zero examples | | Model weights | produced and stored | not visible, cannot be changed | | Definition of the task | the label set sits in the training data | the label set is written into the prompt text | | Format of the output | the procedure's return type, certain | requested through the prompt, must be verified | | Same response to the same input | given by the nature of the procedure | not given, must be measured | | When the number drops | the model is retrained | the scaffolding is changed | The last row is the boundary of responsibility itself. In the training approach the address of a degradation is known: the data or the model. In the using approach the model does not degrade, because the model is not yours; what degrades is either the prompt, or the procedure that reads the output, or a change underneath the component. In the third case only one thing is left in your hands: **having measured the surface**. A measured surface tells you when the change happened; an unmeasured surface tells you only the complaint. The difference between 1.0000 and 0.7778 among the numbers is also read in this frame. The training approach gave the higher number, but in return it asked for 48 labeled examples. The using approach gave 0.7778 **without seeing any labeled example at all**. On a task with a closed label set and plenty of examples, the first wins; when the label set is changeable, examples are expensive, or the task requires producing text, the question flips. What is being compared is not two numbers but **two cost units**. #### Summary - **Two** of the training approach's four settings and **three** of the using approach's four settings move the number by more than 0.0200; counting the seed, four of the using side's five rows do. No item of the two lists is shared. - The training approach's strongest setting is the number of labeled examples: 0.6250 with 8 examples, **1.0000** with 48. The using approach gives **0.7778** with zero labeled examples. - The using approach's strongest setting is the format constraint: **+0.2778** and a **917-token saving**, by far the largest minus sign in the cost column. - Retrying moves the number by **0.0139** and adds two calls and 111 tokens: adding scaffolding does not always improve enough to matter, but its cost is always certain. - The seed, which is out of one's hands, **moves the number by 0.0417** — more than one of the four settings that are in one's hands. - The boundary of responsibility is read in the last row: when the number drops, the training approach changes the model and the using approach changes the scaffolding. #### Next Step In this lesson the seed row appeared once, with a single number, and was left there. Yet that row is the hardest design question of a product: the same user, on the same screen, sending the same text twice can get two different responses. The change of the overall hit rate from 0.7778 to 0.8194 is only the **overall** view of this; in how many records the response really changed and which part of that change is **visible** to the user was never asked. The next lesson counts the two runs of the same input record by record and puts three design decisions that narrow the uncertainty — offering options, imposing a format, grounding — side by side as hit rate and cost. ### Impact on Product Development Source: https://academia.sh/en/courses/introduction-to-ai-engineering/impact-on-product-development Measuring with numbers what it means to design a product around uncertain output: when the same 72 fictional support tickets are called in two runs, on the plain prompt the response text changes in 0.8750 of the records while the decision underneath changes in 0.5972, and in 0.5556 of the records the correct response comes out in only one run. Three design decisions that narrow the uncertainty are measured side by side: imposing a format lowers the visible change from 0.8611 to 0.3056, grounding lowers it to 0.0972 and raises the hit rate to 0.8264 while adding 144 retrieval steps, and offering options raises the change to 0.4306 and doubles the calls. The previous lesson showed the seed row with a single number: the overall hit rate moved from 0.7778 to 0.8194. That row is the hardest design question of a product, and the overall number hides it. The same user, on the same screen, sending the same text twice can get two different responses. Even the overall hit rate staying the same would hide it, because a correct response won on one record may have been lost on another. This lesson counts uncertainty at the record level and separates two questions: **in how many records the output changed** and **what part of that change is visible to the user**. Then the three design decisions that narrow uncertainty are compared on the same measure. - **FR16.** The fiction, the simulator, and the test set carry over unchanged from the previous lesson: 72 records, five topics, seed **20260218**. The second run is taken with seed **20260219**. - **FR17.** What is compared between the two runs is **the output the product shows**. On the plain prompt this is a sentence, on the instructed prompt a single label; the two are counted separately. - **FR18.** The fictional document pool is ten records: tariff descriptions, meter maintenance procedures, and fault codes. The document codes are also fictional. - **FR19.** Retrieval selects the document that overlaps the most with the ticket's tokens. It is a **step**, not a model call; it is counted in a separate column in the cost table. - **FR20.** The hit rate of offering options is read on **a different criterion**: it is enough for the correct label to be among the two candidates. It stands in the same column as the other three rows but does not measure the same thing. - **FR21.** The visible change is good **when it is low**; the hit rate is good when it is high. The two columns point in opposite directions and are read together in the table. - **FR22.** Calls that hit the rate limit land on the same places in both runs because the counter is reset at the start of every measurement; the change count therefore comes only from the model's uncertainty. #### Two Runs of the Same Input The first block sets up the simulator and calls each ticket with two seeds, separating four numbers: did the text change, did the decision change, is it correct in both runs, is it correct in only one. ```python # SIMULATOR. This code does not call a real model endpoint; it is a language model # simulator written with the standard library. The support tickets are also FICTIONAL. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUTPUT = 240, 24 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): tok = [] for s in text.lower().replace(",", " , ").split(): while len(s) > 5: tok.append(s[:5]) s = s[5:] tok.append(s) return tok def trace_id(text): h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["invoice", "fault", "meter", "tariff", "outage"] TEMPLATES = { "invoice": ["the invoice amount this month is very high compared to last month", "i paid my invoice but the system shows a debt", "where can i see the invoice details"], "fault": ["the water is not flowing at home and there is a fault i think", "the connection is broken and it is not working at all", "i opened a fault ticket but the team never came"], "meter": ["the meter reading was entered wrong and the index does not match", "i want an appointment for the meter change", "the index on the meter and the invoice are not the same"], "tariff": ["i want to learn which tariff tier i am on", "what do i need to do for a tariff change", "does the unit price change by subscriber type"], "outage": ["there is an outage in the neighborhood when will it be back", "where is the planned outage announcement tracked", "we have been having outages since the evening"], } EXTRAS = ["the same issue happened last month too", "i am not satisfied with this", "i urgently need information", "i do not have my record number"] TICKETS = [] for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = TEMPLATES[k][int(r() * 3)] if r() < 0.34: m += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": m, "topic": k}) CONCEPTS = TOPICS + ["payment", "complaint", "information", "general"] TRACES = {"invoi": "invoice", "amoun": "invoice", "detai": "invoice", "fault": "fault", "broke": "fault", "worki": "fault", "flowi": "fault", "meter": "meter", "index": "meter", "readi": "meter", "tarif": "tariff", "unit": "tariff", "subsc": "tariff", "tier": "tariff", "outag": "outage", "annou": "outage", "neigh": "outage", "debt": "payment", "paid": "payment", "price": "payment", "satis": "complaint", "issue": "complaint", "urgen": "complaint", "infor": "information", "learn": "information", "where": "information"} TRAPS = {"chang": "tariff", "recor": "fault", "high": "tariff", "shows": "invoice", "team": "outage", "track": "tariff", "enter": "invoice"} FREE_FORM = ["your request seems to relate to {k}", "this falls under {k} , forwarded to the relevant team", "your record has been received , we will review the matter and get back to you"] COUNTER = [0] def call(prompt, seed=SEED, temperature=0.6, max_output=MAX_OUTPUT): COUNTER[0] += 1 tok = tokenize(prompt) if len(tok) > WINDOW: return {"status": "context_overflow", "input": len(tok), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: return {"status": "rate_limit", "input": len(tok), "output": 0, "response": ""} r = rng(seed ^ trace_id(prompt)) if len(tok) > 90 and r() < 0.04: return {"status": "timeout", "input": len(tok), "output": 0, "response": ""} cand = [k for k in CONCEPTS if k[:5] in tok] if len(cand) < 3: cand = CONCEPTS p = {k: 0.15 for k in cand} for t in tok: for table, w in ((TRACES, 1.0), (TRAPS, 1.3)): if table.get(t) in p: p[table[t]] += w if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) else: e = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [v / sum(e) for v in e])] y = k if "only" in tok else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k) out = tokenize(y) if len(out) > max_output: out = out[:max_output] y = " ".join(out) return {"status": "ok", "input": len(tok), "output": len(out), "response": y} def read_label(y): for s in y.lower().split(): if s in TOPICS: return s return None TEST = TICKETS[48:] PLAIN = "Answer this request: " INSTRUCTION = ("Assign the ticket to one of these labels: invoice , fault , meter , tariff , " "outage . Write only the label as the response . Ticket: ") def two_runs(prefix): COUNTER[0] = 0 text_changed = decision_changed = both = one = neither = 0 for x in TEST: a = call(prefix + x["text"], seed=SEED) b = call(prefix + x["text"], seed=SEED + 1) ea, eb = read_label(a["response"]), read_label(b["response"]) text_changed += a["response"] != b["response"] decision_changed += ea != eb d = (ea == x["topic"]) + (eb == x["topic"]) both += d == 2 one += d == 1 neither += d == 0 n = len(TEST) return text_changed / n, decision_changed / n, both / n, one / n, neither / n print(f"{'prompt':<10} {'text changed':>14} {'decision changed':>16} {'both correct':>12} " f"{'only one':>12} {'both wrong':>13}") for name, o in (("plain", PLAIN), ("instructed", INSTRUCTION)): m, k, i2, b1, h = two_runs(o) print(f"{name:<10} {m:>14.4f} {k:>16.4f} {i2:>12.4f} {b1:>12.4f} {h:>13.4f}") ``` ``` prompt text changed decision changed both correct only one both wrong plain 0.8750 0.5972 0.2222 0.5556 0.2222 instructed 0.3472 0.3472 0.6389 0.3056 0.0556 ``` In the first row, the difference between two columns is the main finding of the lesson. On the plain prompt, **the response text changes in 0.8750 of the records**, but **the decision underneath changes in only 0.5972**. The 0.2778 in between is the records where the decision stayed the same although the sentence was composed differently. If what the user sees is the sentence, the product looks half again as indecisive; if what the user sees is the decision, the real indecision is what gets read. **The uncertainty of a product is not the model's uncertainty but the uncertainty of what it puts on the screen.** The fourth column is the most expensive number from the product's point of view. On the plain prompt, in **0.5556 of the records** the correct response comes out in only one run. In these records, whether the user gets a correct response does not depend on anything the user can see or change. On the instructed prompt this share falls to 0.3056 and the "wrong in both runs" share drops from 0.2222 to 0.0556: the instruction does not merely increase the number of correct responses, it also reduces the **persistent errors**. These two rows have three direct consequences for product design. First, sending the same ticket twice is not a bug but expected behavior; storing the response and showing the stored response on the second request brings the visible indecision down to zero without changing the hit rate at all. Second, the "correct in only one" share should be written to the support record: whether a wrong response is persistent or tied to the run can be told apart only if both runs are stored. Third, leaving the text shown to the user free is not free of charge; two different sentences carrying the same decision appear to the user as two different responses and increase the number of complaints independently of the decision itself. #### Three Design Decisions That Narrow Uncertainty The three decisions are compared on the same measure: offering options, imposing a format, grounding. For grounding, a fictional document pool is set up and the document that overlaps the ticket the most is added to the prompt. ```python DOCUMENTS = [("TR-11", "the tariff tier is determined by subscriber type and unit price"), ("TR-12", "tariff change requests are accepted at the start of each period"), ("FT-21", "the invoice amount is calculated from the period reading , " "the detail is on screen"), ("FT-22", "after payment the debt entry is closed the same day"), ("SY-31", "meter index reading is taken each period and goes into the log"), ("SY-32", "meter change is done by appointment , the old index stays in the log"), ("AR-41", "when a fault ticket is opened the field crew is dispatched"), ("AR-42", "for a broken connection the line is checked first"), ("KS-51", "the planned outage announcement is published first"), ("KS-52", "outage follow-up per neighborhood is done on a map")] def retrieve(text): # RETRIEVAL step: the document with the most overlapping pieces b = set(tokenize(text)) return max(DOCUMENTS, key=lambda d: (len(b & set(tokenize(d[1]))), d[0])) def plan(design, x, seed): # prompts and the number of retrieval steps if design == "free": return [(PLAIN + x["text"], seed)], 0 if design == "options": return [(INSTRUCTION + x["text"], seed + j) for j in (0, 2)], 0 if design == "format": return [(INSTRUCTION + x["text"], seed)], 0 k, m = retrieve(x["text"]) return [(INSTRUCTION + "Document " + k + ": " + m + " . Ticket: " + x["text"], seed)], 1 def run(design, seed): COUNTER[0] = 0 shown, correct, tokens, steps = [], 0, 0, 0 for x in TEST: prompts, a = plan(design, x, seed) steps, e, last = steps + a, [], "" for p, sd in prompts: s = call(p, seed=sd) tokens += s["input"] + s["output"] e.append(read_label(s["response"])) last = s["response"] labels = sorted(set(e) - {None}) shown.append(" ".join(labels) if design == "options" else last) correct += (x["topic"] in labels) if design == "options" else (e[0] == x["topic"]) return shown, correct / len(TEST), COUNTER[0], tokens, steps print(f"{'design decision':<18} {'change':>8} {'hit rate':>8} {'calls':>6} " f"{'tokens':>9} {'retrieval':>9}") for name, d in (("no scaffolding", "free"), ("impose format", "format"), ("offer options", "options"), ("grounding", "grounding")): g1, i1, c1, b1, a1 = run(d, SEED) g2, i2, c2, b2, a2 = run(d, SEED + 1) dg = sum(u != v for u, v in zip(g1, g2)) / len(g1) print(f"{name:<18} {dg:>8.4f} {(i1 + i2) / 2:>8.4f} {c1 + c2:>6d} {b1 + b2:>9d} " f"{a1 + a2:>9d}") ``` ``` design decision change hit rate calls tokens retrieval no scaffolding 0.8611 0.5139 144 4921 0 impose format 0.3056 0.7986 144 7514 0 offer options 0.4306 0.9375 288 15013 0 grounding 0.0972 0.8264 144 10835 144 ``` **Imposing a format** is the cheapest decision: it lowers the visible change from 0.8611 to **0.3056**, raises the hit rate from 0.5139 to **0.7986**, and does not touch the call count at all. Its cost is 7514 tokens against 4921 for no scaffolding, and all of the added cost comes from the instruction written into the prompt. **Grounding** is the strongest decision: the visible change drops to **0.0972**, meaning the two runs diverge in fewer than one record in ten. The hit rate is **0.8264**. The reason is that once the document enters the prompt, the evidence for the correct topic outweighs the rest, that is, less of the decision is left to sampling. Its cost has two items: 10835 tokens against 7514 for imposing a format, and **144 retrieval steps**. Retrieval is not a model call but it is a source of latency and carries its own error — when the wrong document arrives, the model gives a **consistently wrong** response. **Offering options** is the row in this table that pays off the second claim of the curriculum. The highest number in the hit rate column belongs to it: 0.9375. But that number is a different criterion — it is enough for the correct label to be among the two candidates, and the choosing is left to the user. On its own criterion, that is **the visible change**, offering options **sets back** imposing a format: from 0.3056 to 0.4306. When one of the two candidates changes, the box on the screen changes too, so the two runs diverge in more records. In return the call count **doubles** and the tokens double. **The added scaffolding made the number worse and its cost definitely rose.** #### Summary - On the plain prompt, the response text changes in **0.8750** of the records, while the decision underneath changes in **0.5972**. The difference is that, depending on what the product puts on the screen, the indecision appears half again as large. - In **0.5556** of the records the correct response comes out in only one run. This share is a loss the user can neither see nor influence; on the instructed prompt it falls to 0.3056, and the persistent wrong share falls from 0.2222 to **0.0556**. - Imposing a format moves the visible change **from 0.8611 to 0.3056** and the hit rate **from 0.5139 to 0.7986** without changing the call count; its cost is 7514 tokens against 4921. - Grounding lowers the change to **0.0972** and makes the hit rate **0.8264**; its cost is 10835 tokens against 7514, and **144 retrieval steps**. When the wrong document arrives, the wrong response becomes stable. - Offering options **raises the visible change from 0.3056 to 0.4306** and doubles the calls; the 0.9375 in the hit rate column is a different criterion and hands the choosing to the user. - The latency class also separates: the no-scaffolding and format-imposing paths are **a single call**, grounding is **a chain of retrieval and call**, offering options is **two parallel calls**. The four rows are read with a single rule: a design decision can be compared only when it is written **on its own criterion** and **with its cost**. #### Next Step Over three lessons the model was treated as a given component, measured from its surface, its settings were counted, and its uncertainty was translated into product language. The words used throughout these three lessons are also a design object. Expressions such as "capability", "understands", "learns", and "general intelligence" are things too different to be written into the same table: some of them can be tied to a test and give a number, and some can never be falsified under any condition. The next lesson separates a set of sentences by this criterion and counts how many can actually be tied to a test. ### The Distinction Between AI and General AI Source: https://academia.sh/en/courses/introduction-to-ai-engineering/ai-and-general-ai-distinction Separating the sentences made about a system by how testable they are: of fourteen fictional claims, five are measurable capability claims and nine are unmeasurable general claims, and one of these carries a number and a direction yet cannot be tested because it has no set. The share that can be tied to a test is 0.3571. Five tests are run on the simulator and four pass: the label hit rate comes out at 0.8264 against a 0.90 threshold and fails, the decision change share passes at 0.0972, the unlabeled response share passes at 0.0139, the document retrieval share passes at 0.9028, and the tokens per request pass at 75.2. The previous three lessons treated the model as a given component, measured it from its surface, counted its settings, and translated its uncertainty into product language. The words used in those three lessons are also a design object. "Capability", "understands", "learns", and "general intelligence" are things too different to be written into the same table: some of them can be tied to a test and give a number, and some can never be falsified under any condition. This lesson ties that separation to a procedure. A series of fictional claim sentences is parsed into four components, the testable ones are selected, and they are **actually run**. What is measured is not the truth of the claims but **how many of them can be tied** to a test. At the end of the lesson one of them is tied, run, and turns out wrong; this shows that making a claim measurable does not make it true. - **FR23.** A claim counts as **testable** when it carries four components: a **task**, a **metric**, a **set**, and a **threshold** together with its direction. If one of the four is missing, the claim counts as general. This is a decision and it is held fixed throughout the lesson. - **FR24.** The components are read from the tokens of the sentence; the classification is not done by hand, it is a running procedure. The tokenizer carries over unchanged from the previous lessons. - **FR25.** The claim sentences are **fiction** and none of them is about a product, a model, or an institution; all of them are about the fictional support assistant built in the previous three lessons. - **FR26.** The system under test is the strongest scaffolding of the previous lesson: a grounded instructed call. The measurement is made on the 72-record test set and in two runs. - **FR27.** A test is run only when the classification finds the claim measurable. No test is written for general claims, because there is no test to write. - **FR28.** The direction of the threshold is part of the claim. A metric without a threshold gives a number but makes no **decision**; a claim passes or fails only together with its direction. #### When a Claim Is Testable The sentence "the system understands support tickets" can never be falsified, because it does not say on which task, on which set, with which number, and at which threshold it will count as understood. The sentence "label hit rate on the 72 record set is above 0.90", on the other hand, can be falsified. The difference between the two is not truth but **tie-ability**. ```python # SIMULATOR. This code does not call a real model endpoint; it is a language model # simulator written with the standard library. The support tickets and claims are also # FICTIONAL. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUTPUT = 240, 24 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): tok = [] for s in text.lower().replace(",", " , ").split(): while len(s) > 5: tok.append(s[:5]) s = s[5:] tok.append(s) return tok def trace_id(text): h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["invoice", "fault", "meter", "tariff", "outage"] TEMPLATES = { "invoice": ["the invoice amount this month is very high compared to last month", "i paid my invoice but the system shows a debt", "where can i see the invoice details"], "fault": ["the water is not flowing at home and there is a fault i think", "the connection is broken and it is not working at all", "i opened a fault ticket but the team never came"], "meter": ["the meter reading was entered wrong and the index does not match", "i want an appointment for the meter change", "the index on the meter and the invoice are not the same"], "tariff": ["i want to learn which tariff tier i am on", "what do i need to do for a tariff change", "does the unit price change by subscriber type"], "outage": ["there is an outage in the neighborhood when will it be back", "where is the planned outage announcement tracked", "we have been having outages since the evening"], } EXTRAS = ["the same issue happened last month too", "i am not satisfied with this", "i urgently need information", "i do not have my record number"] TICKETS = [] for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = TEMPLATES[k][int(r() * 3)] if r() < 0.34: m += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": m, "topic": k}) CONCEPTS = TOPICS + ["payment", "complaint", "information", "general"] TRACES = {"invoi": "invoice", "amoun": "invoice", "detai": "invoice", "fault": "fault", "broke": "fault", "worki": "fault", "flowi": "fault", "meter": "meter", "index": "meter", "readi": "meter", "tarif": "tariff", "unit": "tariff", "subsc": "tariff", "tier": "tariff", "outag": "outage", "annou": "outage", "neigh": "outage", "debt": "payment", "paid": "payment", "price": "payment", "satis": "complaint", "issue": "complaint", "urgen": "complaint", "infor": "information", "learn": "information", "where": "information"} TRAPS = {"chang": "tariff", "recor": "fault", "high": "tariff", "shows": "invoice", "team": "outage", "track": "tariff", "enter": "invoice"} FREE_FORM = ["your request seems to relate to {k}", "this falls under {k} , forwarded to the relevant team", "your record has been received , we will review the matter and get back to you"] COUNTER = [0] def call(prompt, seed=SEED, temperature=0.6, max_output=MAX_OUTPUT): COUNTER[0] += 1 tok = tokenize(prompt) if len(tok) > WINDOW: return {"status": "context_overflow", "input": len(tok), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: return {"status": "rate_limit", "input": len(tok), "output": 0, "response": ""} r = rng(seed ^ trace_id(prompt)) if len(tok) > 90 and r() < 0.04: return {"status": "timeout", "input": len(tok), "output": 0, "response": ""} cand = [k for k in CONCEPTS if k[:5] in tok] if len(cand) < 3: cand = CONCEPTS p = {k: 0.15 for k in cand} for t in tok: for table, w in ((TRACES, 1.0), (TRAPS, 1.3)): if table.get(t) in p: p[table[t]] += w if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) else: e = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [v / sum(e) for v in e])] y = k if "only" in tok else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k) out = tokenize(y) if len(out) > max_output: out = out[:max_output] y = " ".join(out) return {"status": "ok", "input": len(tok), "output": len(out), "response": y} def read_label(y): for s in y.lower().split(): if s in TOPICS: return s return None CLAIMS = [("label hit rate on the 72 record set is above 0.90", "hit_rate", 0.90, ">"), ("the system understands support tickets", "", 0, ""), ("the share of changed decisions on the 72 record set is below 0.10", "change", 0.10, "<"), ("the system is a smart assistant", "", 0, ""), ("the system always produces the correct label", "", 0, ""), ("the system has human level language understanding", "", 0, ""), ("the share of responses without a label on the 72 record set is below 0.05", "unlabeled", 0.05, "<"), ("the system has general intelligence", "", 0, ""), ("the system stays below 140 tokens per request", "", 0, ""), ("the share of correct documents retrieved from the vocabulary is above 0.80", "retrieval", 0.80, ">"), ("the system keeps learning", "", 0, ""), ("tokens per request on the 72 record set is below 140", "tokens", 140, "<"), ("the system produces creative solutions", "", 0, ""), ("the system gets smarter as it grows", "", 0, "")] TASK_KEYS = ("label", "decis", "answe", "retri", "reque") METRIC_KEYS = ("hit", "share", "token", "call", "step") SET_KEYS = ("recor", "set", "vocab", "examp") DIRECTION_KEYS = ("above", "below") def matches(pieces, keys): return any(t.startswith(k) for t in pieces for k in keys) def classify(c): b = tokenize(c) return (matches(b, TASK_KEYS), matches(b, METRIC_KEYS), matches(b, SET_KEYS), matches(b, DIRECTION_KEYS) and any(any(d.isdigit() for d in t) for t in b)) print(f"{'claim':<76} {'task':>5} {'metric':>6} {'set':>4} {'threshold':>9} {'class':>11}") CHECKS = [] for c, metric, threshold, direction in CLAIMS: g, o, k, e = classify(c) if g and o and k and e: CHECKS.append((metric, threshold, direction)) print(f"{c:<76} {'+' if g else '-':>5} {'+' if o else '-':>6} {'+' if k else '-':>4} " f"{'+' if e else '-':>9} {'measurable' if g and o and k and e else 'general':>11}") numeric = sum(1 for c, *_ in CLAIMS if classify(c)[3]) print(f"\ncan be tied to a test {len(CHECKS)}/{len(CLAIMS)} = {len(CHECKS) / len(CLAIMS):.4f}, " f"carries a number and direction yet cannot be tied {numeric - len(CHECKS)}") ``` ``` claim task metric set threshold class label hit rate on the 72 record set is above 0.90 + + + + measurable the system understands support tickets - - - - general the share of changed decisions on the 72 record set is below 0.10 + + + + measurable the system is a smart assistant - - - - general the system always produces the correct label + - - - general the system has human level language understanding - - - - general the share of responses without a label on the 72 record set is below 0.05 + + + + measurable the system has general intelligence - - - - general the system stays below 140 tokens per request + + - + general the share of correct documents retrieved from the vocabulary is above 0.80 + + + + measurable the system keeps learning - - - - general tokens per request on the 72 record set is below 140 + + + + measurable the system produces creative solutions - - - - general the system gets smarter as it grows - - - - general can be tied to a test 5/14 = 0.3571, carries a number and direction yet cannot be tied 1 ``` Of the fourteen sentences, **five** can be tied to a test; the share is **0.3571**. Of the remaining nine, eight carry none of the four components or only one. The fifth row is the most instructive of them: "the system always produces the correct label" carries a task name but gives no metric, no set, and no threshold. "Always" is not a threshold; because it is not written on which set it will be tested, a single counterexample refutes it, and for that reason nobody tests it. The number in the last line gives the real distinction: **a claim can carry a number and still be untestable**. The ninth sentence says "the system stays below 140 tokens per request"; it has a task, a metric, a threshold, and a direction, but **no set**. Because it is not written on which tickets, with which scaffolding, and with prompts of which length it was measured, the number can be produced to take the desired value. The twelfth row adds the set to the same claim and the sentence becomes testable. The only difference between them is the phrase **on the 72-record set**. #### The Tests Are Run The classification says that a sentence is testable, not that it is true. The second block runs five tests on the strongest scaffolding of the previous lesson. ```python DOCUMENTS = [("TR-11", "the tariff tier is determined by subscriber type and unit price"), ("TR-12", "tariff change requests are accepted at the start of each period"), ("FT-21", "the invoice amount is calculated from the period reading , " "the detail is on screen"), ("FT-22", "after payment the debt entry is closed the same day"), ("SY-31", "meter index reading is taken each period and goes into the log"), ("SY-32", "meter change is done by appointment , the old index stays in the log"), ("AR-41", "when a fault ticket is opened the field crew is dispatched"), ("AR-42", "for a broken connection the line is checked first"), ("KS-51", "the planned outage announcement is published first"), ("KS-52", "outage follow-up per neighborhood is done on a map")] CODES = {"TR": "tariff", "FT": "invoice", "SY": "meter", "AR": "fault", "KS": "outage"} TEST = TICKETS[48:] INSTRUCTION = ("Assign the ticket to one of these labels: invoice , fault , meter , tariff , " "outage . Write only the label as the response . Ticket: ") def retrieve(text): b = set(tokenize(text)) return max(DOCUMENTS, key=lambda d: (len(b & set(tokenize(d[1]))), d[0])) def run(seed): COUNTER[0] = 0 labels, tokens, docs = [], 0, 0 for x in TEST: k, m = retrieve(x["text"]) docs += CODES[k[:2]] == x["topic"] s = call(INSTRUCTION + "Document " + k + ": " + m + " . Ticket: " + x["text"], seed=seed) tokens += s["input"] + s["output"] labels.append(read_label(s["response"])) return labels, tokens, docs / len(TEST) e1, b1, doc_rate = run(SEED) e2, b2, _ = run(SEED + 1) n = len(TEST) MEASURES = {"hit_rate": sum((a == x["topic"]) + (b == x["topic"]) for a, b, x in zip(e1, e2, TEST)) / (2 * n), "change": sum(a != b for a, b in zip(e1, e2)) / n, "unlabeled": sum((a is None) + (b is None) for a, b in zip(e1, e2)) / (2 * n), "retrieval": doc_rate, "tokens": (b1 + b2) / (2 * n)} print(f"{'test':<12} {'threshold':>12} {'measured':>10} {'result':>8}") passed = 0 for metric, threshold, direction in CHECKS: v = MEASURES[metric] ok = v > threshold if direction == ">" else v < threshold passed += ok b = "{:.4f}" if threshold < 1 else "{:.1f}" print(f"{metric:<12} {direction + ' ' + b.format(threshold):>12} {b.format(v):>10} " f"{'passed' if ok else 'failed':>8}") print(f"\ntested {len(CHECKS)}, passed {passed}, failed {len(CHECKS) - passed}; " f"tested among general claims 0") ``` ``` test threshold measured result hit_rate > 0.9000 0.8264 failed change < 0.1000 0.0972 passed unlabeled < 0.0500 0.0139 passed retrieval > 0.8000 0.9028 passed tokens < 140.0 75.2 passed tested 5, passed 4, failed 1; tested among general claims 0 ``` Four of the five tests passed, one failed. The first row is the most important result of the lesson: the claim said **0.90**, the measurement gave **0.8264**. The claim is wrong. A measurable claim can turn out wrong; **what makes it valuable is not that it is true but that it can be wrong**. The third row shows the other side: the claim that the unlabeled response share stays below 0.05 passed at **0.0139**, and a claim that passes is also a result, not a courtesy. The measurement covered **the whole surface**, including the calls that hit the rate limit and produced no response, and the claim still held. The last line is the silent result of the lesson. Among the general claims, the number tested is **zero**. Nothing could be said about nine sentences: they neither passed nor failed. The claim that a system "has general intelligence" or "gets smarter as it develops" cannot be discussed, because no counterexample can be produced for it; it has no place in an engineering document, because it changes no decision. **The term general artificial intelligence (general AI) is not a capability class but a lack of tests.** A sentence that wants to say what a system does has to write its task, its set, its metric, and its threshold; when it does not write them, what it says cannot be measured. #### Summary - A claim becomes testable with four components: a task, a metric, a set, and a threshold together with its direction. Of the fourteen fictional sentences, **five** carry these four; the share is **0.3571**. - Carrying a number is not enough. The sentence "the system stays below 140 tokens per request" carries a task, a metric, and a threshold but **has no set**; when "on the 72-record set" is added to the same sentence, it becomes testable. - Five tests were run: **four passed, one failed**. The decision change share of 0.0972, the unlabeled response share of 0.0139, the document retrieval share of 0.9028, and the tokens per request of 75.2 passed their thresholds. - A measurable claim can also turn out wrong: the label hit rate claim said 0.90, the measurement gave **0.8264**. The measurement covers the whole surface, rate limit included. - Among the general claims, the number tested is **zero**. The term general AI is not a capability class but a lack of tests; in an engineering document it changes no decision. #### Next Step In four lessons the model was always looked at **from the outside**. Its measure was set up, who holds its settings was counted, its uncertainty was translated into product language, and the sentences made about it were separated by testability. All of this was done on the surface of a closed box, and that was a deliberate choice. But every number measured on the surface has a reason, and that reason is inside: why 0.8264 is not 1.0000, why "meter replacement" and "tariff change" get confused, why a seven-word ticket makes twelve tokens, why the window ends at 240. The next topic opens with these questions: what is the **unit** the model works in, how does it use distant context, where does its limit come from, and in which steps is a response produced. ## How Language Models Work Measuring the surface a given language model exposes to the outside: the number of tokens the same Turkish text costs and how its agglutinative structure enlarges it, the cost of attention being able to use distant context as a number of comparisons, how many examples it takes for a task set to improve in pretraining and in fine-tuning, what the most likely choice and sampled choice give in two runs on the same prompt, what dropping from the head, dropping from the middle and summarizing gain in tokens and lose in hit rate when the context window is exceeded, the collapse of the bare call on counting and exact arithmetic tasks, the fabrication rate of claims not based on a document in an ungrounded response and the cost of three mitigations, the comparison of open- and closed-weight access on three metrics, how model selection changes as the metric weight changes, and the token and hit rate comparison of four scaffoldings. ### Tokens Source: https://academia.sh/en/courses/introduction-to-ai-engineering/tokens Measuring the input unit of text for the model as a unit of cost: 300 fictional support tickets come to 4,674 words and 8,334 tokens, 1.7831 tokens per word; with suffix pieces removed 6,176 tokens remain and the suffix share comes out at 0.2589. Six roots take 6 tokens in plain form and 30 tokens as plural and possessive suffixes pile on. A bare call gives 27.78 tokens per ticket and a field hit rate of 0.9722, while the scaffolding that drops frequent pieces keeps the same hit rate with 11.68 tokens at the 0.50 threshold and falls to 6.23 tokens at 0.30, but the hit rate drops to 0.8611. The Field and Roles topic looked at the model from the outside across four lessons: who builds what, how many settings the training approach and the using approach have in hand, what the same input gives in two runs, and which claim could be tied to a test. In all four the model was a box and the lid of the box was never opened. What is inside, which unit it works with and where its limit comes from were never asked. This topic opens with that question and starts from the very bottom. Sending text to a language model is not really sending text: the text is first split into pieces, each piece is turned into a number, and the model sees only that sequence of numbers. This piece is called a **token**. A token is not a technical detail; it is the **unit of cost** of this curriculum: the window is measured in tokens, the fee is counted in tokens, latency grows with the number of generated tokens. The rule of the course works here for the first time — a given component is measured from its **surface**, and the surface this lesson measures is the token count. - **LM1.** The support tickets are **fictional**: they are generated inside the lesson, seed 20260218, and come from no source system. The set is 300 short Turkish tickets. - **LM2.** The model is a **simulator**: it is written inside the lesson with the standard library, and no real model endpoint is called. The surface it carries is tokenization, token counting, the context window, the stop condition, the maximum output length and seed-dependent uncertainty. - **LM3.** The tokenizer is fictional too: a vocabulary of 177 pieces and leftmost longest match. The tokenization procedure, the vocabulary and the out-of-vocabulary concepts were measured in the Natural Language Processing and Computer Vision course; the procedure is **not repeated** here, it is only read as a unit of cost. - **LM4.** A sequence with no counterpart in the vocabulary falls to single characters. In this simulator there is no out-of-vocabulary case; the cost of the unknown is paid as a token count. - **LM5.** Every ticket has a **field triple**: object, status, request. The triple is known from the fiction itself; the reader procedure takes the first matching root in the token sequence. - **LM6.** The measure is the same on every row: **tokens per ticket** (smaller is better) and **field hit rate** (larger is better). - **LM7.** In this lesson the **bare call** is sending the ticket as it is; the scaffolding is every operation applied to the text before it is sent. - **LM8.** The input token count is independent of the run: the same text gives the same number in every run. The output length is not independent. #### The Input Unit of Text for the Model The simulator's tokenizer carries a small vocabulary of pieces: the frequent roots of the field, frequent suffixes and single characters. A word is scanned from the left, at each step the longest piece found in the vocabulary is taken, and if none is found a single character is taken. Punctuation is a token on its own. ```python # SIMULATOR — in this course the model is a simulator written in the lesson with # the standard library; no real model endpoint is called. The support tickets # are FICTIONAL and generated in the lesson, seed 20260218. SEED, M32, TICKETS = 20260218, 0xFFFFFFFF, 300 ROOTS = """sayac kapak muhur vana abone adres fatura tarife odeme borc ariza kod okuma olcum servis ekip ziyaret kayit talep destek hat bakim birim gun ay yil saat numara bilgi islem sistem hata sonuc deger endeks sozlesme kampanya indirim gecikme faiz itiraz basvuru sikayet cevap bekle gonder geri ac kapa oku yaz bul bak gel ver yap don calis bozuk kirik eksik normal yuksek dusuk yeni eski acik kapali lutfen merhaba tesekkur iyi rica eder mumkun tekrar hala durum konu ile icin ve bir bu ne var yok ama kadar sonra once ben biz siz""".split() SUFFIXES = """lari leri lar ler nin nun den dan de da te ta iniz unuz imiz umuz in un im um si su li lu di du ti tu miz niz yor mis dir tir cek cak ki ya ye yi yu ma me ni nu la le i u e a""".split() PIECES = set(ROOTS) | set(SUFFIXES) | set("abcdefghijklmnopqrstuvwxyz0123456789") PUNCT = ".,:;?-/" def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def split_pieces(s): p, i = [], 0 while i < len(s): for j in range(min(len(s), i + 9), i, -1): if s[i:j] in PIECES: p.append(s[i:j]) i = j break else: p.append(s[i]) i += 1 return p def tokenize(text): out, cur = [], "" for c in text.lower(): if c.isalnum(): cur += c else: if cur: out += split_pieces(cur) cur = "" if c in PUNCT: out.append(c) return out + (split_pieces(cur) if cur else []) OBJECTS = {"fatura": ("faturamizin", ["yuksek", "yuksek", "eksik"]), "tarife": ("tarifemizin", ["yuksek", "eski", "yuksek"]), "endeks": ("endeksimizin", ["yuksek", "dusuk", "eksik"]), "sayac": ("sayacimizin", ["bozuk", "kirik", "eski"]), "kapak": ("kapaginin", ["kirik", "eksik", "acik"]), "vana": ("vananin", ["kapali", "bozuk", "acik"]), "hat": ("hattimizin", ["kapali", "bozuk", "eski"]), "odeme": ("odememizin", ["eksik", "gecikmeli", "yuksek"])} OBJECT_WEIGHTS = [0.20, 0.14, 0.12, 0.14, 0.10, 0.09, 0.11, 0.10] REQUESTS = {"gonder": "ekip gonderin", "ac": "kayit acin", "ver": "bilgi verin", "don": "geri donun"} TEMPLATES = ["{a} {n} durumu {d} ve bu konuda hala bir cevap yok. {i} lutfen. {k}.", "{a} {n} durumu {d} ve tekrar yazdim ama sonuc yok. {i} lutfen. {k}.", "{a} {n} durumu {d} ve bu ay ikinci basvurum. {i} lutfen. {k}."] OPENINGS = ["Merhaba,", "Iyi gunler,", "Merhaba,", "Iyi gunler,"] CLOSINGS = ["Tesekkur ederim", "Rica ederim", "Tesekkur ederim", "Iyi gunler"] OBJECT_NAMES, REQUEST_ROOTS = list(OBJECTS), list(REQUESTS) DATA = [] for i in range(TICKETS): r = rng(SEED + i) n = OBJECT_NAMES[pick(r(), OBJECT_WEIGHTS)] long_name, statuses = OBJECTS[n] d = statuses[int(r() * 3)] k = REQUEST_ROOTS[int(r() * 4)] text = TEMPLATES[int(r() * 3)].format(a=OPENINGS[int(r() * 4)], n=long_name, d=d, i=REQUESTS[k], k=CLOSINGS[int(r() * 4)]) DATA.append({"text": text, "field": (n, tokenize(d)[0], k)}) print(f"{TICKETS} fictional support tickets, simulator vocabulary {len(PIECES)} pieces") for i in (0, 2, 7): b = tokenize(DATA[i]["text"]) print(f"\n{DATA[i]['text']}") print(f" {len(DATA[i]['text'].split())} words, {len(b)} tokens") print(" " + " ".join(b[:12]) + " ...") ``` ``` 300 fictional support tickets, simulator vocabulary 177 pieces Iyi gunler, odememizin durumu gecikmeli ve tekrar yazdim ama sonuc yok. geri donun lutfen. Iyi gunler. 16 words, 29 tokens iyi gun ler , odeme miz in durum u gecikme li ve ... Merhaba, tarifemizin durumu yuksek ve bu konuda hala bir cevap yok. kayit acin lutfen. Iyi gunler. 16 words, 26 tokens merhaba , tarife miz in durum u yuksek ve bu konu da ... Iyi gunler, vananin durumu bozuk ve bu konuda hala bir cevap yok. ekip gonderin lutfen. Tesekkur ederim. 17 words, 27 tokens iyi gun ler , vana nin durum u bozuk ve bu konu ... ``` A sixteen-word ticket comes to 29 tokens. The gap does not come from a single place: punctuation is counted separately, `odememizin` (of our payment) is split into three pieces, `gecikmeli` (delayed) into two. Estimating the length of a text by words is, in this unit, counting systematically short. #### The Cost of Agglutinative Structure Turkish is an agglutinative language: successive suffixes can pile onto a root so that a whole sentence is built in a single word. For the reader this is brevity, for the one counting tokens it is cost. Six roots are measured across six inflection steps. ```python FORMS = { "sayac": "sayac sayaclar sayacimiz sayacimizin sayaclarimizin sayaclarimizdaki", "kapak": "kapak kapaklar kapagimiz kapaginin kapaklarimizin kapaklarimizdaki", "fatura": "fatura faturalar faturamiz faturamizin faturalarimizin faturalarimizdaki", "odeme": "odeme odemeler odememiz odememizin odemelerimizin odemelerimizdeki", "tarife": "tarife tarifeler tarifemiz tarifemizin tarifelerimizin tarifelerimizdeki", "abone": "abone aboneler abonemiz abonemizin abonelerimizin abonelerimizdeki", } HEADERS = ("plain", "pl", "poss", "poss+", "pl+", "pl++") print(f"{'root':<8} " + " ".join(f"{b:>5}" for b in HEADERS)) total = [0] * 6 for k, s in FORMS.items(): n = [len(tokenize(b)) for b in s.split()] total = [a + b for a, b in zip(total, n)] print(f"{k:<8} " + " ".join(f"{x:>5}" for x in n)) print(f"{'total':<8} " + " ".join(f"{x:>5}" for x in total)) words = sum(len(x["text"].split()) for x in DATA) tokens = sum(len(tokenize(x["text"])) for x in DATA) root_tokens = sum(len([b for b in tokenize(x["text"]) if b not in SUFFIXES]) for x in DATA) print(f"\n{TICKETS} tickets: {words} words, {tokens} tokens, ratio {tokens / words:.4f}") print(f"with suffix pieces removed {root_tokens} tokens, ratio {root_tokens / words:.4f}") print(f"suffix share {(tokens - root_tokens) / tokens:.4f}") ``` ``` root plain pl poss poss+ pl+ pl++ sayac 1 2 2 3 4 5 kapak 1 2 3 4 4 5 fatura 1 2 2 3 4 5 odeme 1 2 2 3 4 5 tarife 1 2 2 3 4 5 abone 1 2 2 3 4 5 total 6 12 13 19 24 30 300 tickets: 4674 words, 8334 tokens, ratio 1.7831 with suffix pieces removed 6176 tokens, ratio 1.3214 suffix share 0.2589 ``` Six roots take 6 tokens in plain form and 30 tokens fully inflected: the same six concepts grow fivefold. The word count never changed; the only thing that changed is the suffixes. The `kapak` (cover) row shows one more detail: in the form `kapagimiz` the consonant softens, so the root no longer matches the vocabulary piece `kapak` and one extra token is paid. At the dataset level the ratio is 1.7831: each word comes to 1.78 tokens on average. When suffix pieces are dropped from the sequence, 8,334 tokens fall to 6,176; **the suffix share is 0.2589**. Writing the same information in a language that is not agglutinative would largely erase this share. This is not a judgment about the language but a **budget fact**: the same window takes in less of a Turkish text. #### When the Same Information Is Shortened The information a ticket carries is three fields: object, status, request. Courtesy phrases, conjunctions and repeated sentences do not change these three, yet they pay tokens. A scaffolding can be built: drop the pieces whose **document frequency** exceeds a threshold. A frequent piece does not discriminate; a rare one does. ```python DOCUMENTS = [tokenize(x["text"]) for x in DATA] df = {} for b in DOCUMENTS: for t in set(b): df[t] = df.get(t, 0) + 1 / TICKETS STATUS_ROOTS = {tokenize(d)[0] for _, ds in OBJECTS.values() for d in ds} def read_field(b): def first(names): return next((t for t in b if t in names), None) return (first(set(OBJECT_NAMES)), first(STATUS_ROOTS), first(set(REQUEST_ROOTS))) def shorten(b, threshold): return [t for t in b if df[t] < threshold] def call(tokens, seed, max_output=40): """The simulator generates step by step; it stops when it reaches the stop condition or the maximum output length. The output length depends on the seed.""" r, base, n_out = rng(seed + len(tokens)), 6 + len(tokens) // 3, 0 while n_out < max_output and (n_out < base or r() >= 0.32): n_out += 1 return {"input": len(tokens), "output": n_out, "stop": "limit" if n_out == max_output else "condition"} def measure(name, lists): correct = sum(sum(a == b for a, b in zip(read_field(b), DATA[i]["field"])) for i, b in enumerate(lists)) n = sum(len(b) for b in lists) print(f"{name:<28} {n:>6} {n / TICKETS:>8.2f} {correct / (3 * TICKETS):>8.4f}") print(f"{'scaffolding':<28} {'tokens':>6} {'ticket':>8} {'field':>8}") measure("bare call", DOCUMENTS) measure("drop frequent pieces (0.50)", [shorten(b, 0.50) for b in DOCUMENTS]) measure("drop frequent pieces (0.30)", [shorten(b, 0.30) for b in DOCUMENTS]) k = shorten(DOCUMENTS[2], 0.50) print("\n" + " ".join(DOCUMENTS[2])) print(" ".join(k)) for t in (SEED, SEED + 1): y = call(k, t) print(f"seed {t} input {y['input']} tokens output {y['output']} tokens" f" stop {y['stop']}") ``` ``` scaffolding tokens ticket field bare call 8334 27.78 0.9722 drop frequent pieces (0.50) 3505 11.68 0.9722 drop frequent pieces (0.30) 1868 6.23 0.8611 merhaba , tarife miz in durum u yuksek ve bu konu da hala bir cevap yok . kayit ac in lutfen . iyi gun ler . merhaba tarife yuksek konu da hala bir cevap kayit ac seed 20260218 input 10 tokens output 11 tokens stop condition seed 20260219 input 10 tokens output 10 tokens stop condition ``` The triple is read from here. The **bare call** spends 27.78 tokens per ticket and its field hit rate is 0.9722. The **scaffolding** drops to 11.68 tokens per ticket at the 0.50 threshold — a gain of 0.5794 — and does **not lower** the hit rate at all: all sixteen dropped pieces are courtesy phrases, conjunctions or inflection suffixes. The **cost** is a preprocessing step, a frequency table and a loss of readability; because the number does not fall, this scaffolding pays off. When the threshold is pulled to 0.30 the picture turns. Tokens fall to 6.23, the gain becomes 0.7759, but the hit rate drops to 0.8611: the document frequency of the `yuksek` (high) status is 0.3333, and because it stays above the threshold it is dropped. The most frequent status is also the most asked-about status. **Adding scaffolding does not always improve things**, and whether it improves them is seen only when it is written together with its cost. The bare call staying at 0.9722 is also a finding. All 25 missed fields belong to the `kapak` (cover) object: because the root softens in the form `kapaginin` (of its cover), the `kapak` piece never appears in the token sequence. Agglutinative structure does not only inflate the count; it can also **erase** the root from the token stream. The last two lines separate the uncertainty. When the same prompt is called with two different seeds, the input token count is 10 in both — tokenization is a procedure, independent of the run. The output is 11 and 10, and both ended with the stop condition. Of the two quantities countable on the model's surface, one is independent of the run and the other is not; the budget calculation is built on this distinction. #### Summary - A token is the input unit of text for the model and the unit of cost of this curriculum. The simulator applies leftmost longest match with a vocabulary of 177 pieces; the tickets are fictional, seed 20260218. - The same 300 tickets come to 4,674 words and 8,334 tokens: 1.7831 per word. An estimate made from the word count systematically counts short in this unit. - Agglutinative structure is a measurable cost: six roots come to 6 tokens in plain form and 30 tokens fully inflected, and at the dataset level the suffix share is 0.2589. - The bare call gives 27.78 tokens per ticket and a field hit rate of 0.9722. The scaffolding that drops frequent pieces at the 0.50 threshold falls to 11.68 tokens and keeps the hit rate; the 0.30 threshold falls to 6.23 tokens but lowers the hit rate to 0.8611. - The input token count is independent of the run, the output length is not: the same prompt gives 10 input tokens and 11 and 10 output tokens on two seeds. #### Next Step The text was split into tokens and the cost of every token was counted. There is still only a sequence in the middle: what the model does with this sequence, how a token is related to another one a hundred tokens earlier was not asked. The next lesson covers conceptually why the **attention** mechanism can use distant context and counts its cost: how the number of comparisons grows as the context lengthens, and at what distance a cheap procedure content with a fixed window collapses. ### Transformer Intuition Source: https://academia.sh/en/courses/introduction-to-ai-engineering/transformer-intuition Measuring why attention can use distant context and what it costs as a number of comparisons: in 240 fictional ticket streams the topic object is on average 63.9 tokens from the end, and the reader that looks at the last 16 tokens gives a hit rate of 0.0000, returning the wrong object in 83 streams and nothing in 157. The last 64 tokens rise to 0.4917 but stay at 0.0000 beyond 65 tokens; the weight that decays with distance stays at 0.3333 although it sees every position. The weight independent of distance gives 1.0000 and its cost is 70.9 comparisons: in a full pass, when the context doubles the comparisons quadruple, and at 1024 tokens it is 8.26 times the fixed window. The previous lesson split text into tokens and counted that each token has a cost. There is a sequence in hand, but what the model does with it was not asked. In a support stream the topic is stated in the first sentence and referred to in the last sentence only as "this matter": across the fifty tokens in between, the link has to be carried without breaking. The structure of the transformer architecture, attention and the embedding layer was built and measured in the Architectures topic of the Deep Learning course; **the architecture is not repeated here**. This lesson takes up a single question from the user's side: why can attention use distant context and what does it cost. The answer hides in a single structural detail — in attention the weight of a position is **not tied to its distance** — and the cost in a single number: the **number of comparisons**. - **LM9.** The ticket streams are **fictional**: they are generated inside the lesson, seed 20260218. The dataset is 240 streams; each stream is a topic sentence, two to nine filler sentences and a closing question. - **LM10.** The tokenizer is the same procedure as in the previous lesson: a vocabulary of pieces and leftmost longest match. - **LM11.** The task is a single one: finding the **topic object** of the stream. The correct answer is the object that appears in the first sentence. - **LM12.** In a part of the streams a **second object** appears near the end. This second object is not the topic; it misleads every procedure that looks at proximity. - **LM13.** The match score in the simulator is **given by hand**: object root 1.00, status root 0.25, the rest 0.02. In a real model this score is learned. What is measured is not where the score comes from but **how many positions are compared**. - **LM14.** Readers differ only in coverage: the fixed window looks at the last k positions, the decaying reader looks at every position but multiplies the weight by a decay with distance, attention looks at every position **independently of distance**. - **LM15.** The measure is the same on every row: topic object hit rate (larger is better) and number of comparisons (smaller is better). - **LM16.** The number of comparisons is independent of the run; it is a counted quantity, not a measured one. #### Link Distance The streams vary a single variable in a controlled way: the distance of the topic object from the end of the sequence. ```python # SIMULATOR — the model is a simulator written in the lesson with the standard # library; no real endpoint is called. Ticket streams are FICTIONAL, seed 20260218. SEED, M32, STREAMS = 20260218, 0xFFFFFFFF, 240 ROOTS = """meter seal valve line invoice tariff reading payment customer address fault code service team visit record ticket support repair day month year hour number detail process system error result value late objection application reply wait send back open close read write find look come give do want work broken cracked missing normal high low new old closed please hello thanks good request again still status matter week same two every problem with for and a the this that what is of our no there but did got wrote opened came applied asked waited through nothing answer too also as well about i ago""".split() SUFFIXES = """s es ed ing er ly""".split() PIECES = set(ROOTS) | set(SUFFIXES) | set("abcdefghijklmnopqrstuvwxyz0123456789") def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def tokenize(text): out, cur = [], "" for c in text.lower() + " ": if c.isalnum(): cur += c continue i = 0 while i < len(cur): for j in range(min(len(cur), i + 9), i, -1): if cur[i:j] in PIECES: out.append(cur[i:j]) i = j break else: out.append(cur[i]) i += 1 cur = "" if c in ".,?": out.append(c) return out OBJECTS = {"invoice": ("our invoice", ["high", "missing"]), "tariff": ("our tariff", ["high", "old"]), "reading": ("our reading", ["high", "low"]), "meter": ("our meter", ["broken", "old"]), "seal": ("our seal", ["cracked", "missing"]), "valve": ("our valve", ["closed", "broken"]), "line": ("our line", ["closed", "old"]), "payment": ("our payment", ["missing", "late"])} FILLERS = ["I wrote again about this but got no reply.", "I still waited and there is no result.", "I opened a ticket this month too.", "The service team came but did nothing.", "There is no result with the record number.", "I applied through the system but no reply came.", "I wrote again two days ago.", "I asked for details about the same status every week."] OBJECT_NAMES = list(OBJECTS) DATA = [] for i in range(STREAMS): r = rng(SEED + i) n = OBJECT_NAMES[int(r() * 8)] long_name, statuses = OBJECTS[n] lines = [f"Hello, the status of {long_name} is {statuses[int(r() * 2)]}."] j = int(r() * 8) for _ in range(2 + int(r() * 8)): lines.append(FILLERS[j % 8]) j += 3 distractor = None if r() < 0.35: distractor = OBJECT_NAMES[(OBJECT_NAMES.index(n) + 1 + int(r() * 7)) % 8] lines.append(f"There is also a record for the {distractor}.") lines.append("What is the status of this matter?") seq = tokenize(" ".join(lines)) DATA.append({"seq": seq, "object": n, "distractor": distractor, "distance": len(seq) - 1 - seq.index(n)}) dist = [x["distance"] for x in DATA] print(f"{STREAMS} fictional ticket streams; {sum(x['distractor'] is not None for x in DATA)}" f" streams contain a second object") print(f"stream length {min(len(x['seq']) for x in DATA)}" f"-{max(len(x['seq']) for x in DATA)} tokens, mean" f" {sum(len(x['seq']) for x in DATA) / STREAMS:.1f}") print(f"distance of the topic object from the end {min(dist)}-{max(dist)}, mean {sum(dist) / STREAMS:.1f}") print("\n" + " ".join(DATA[3]["seq"])) ``` ``` 240 fictional ticket streams; 83 streams contain a second object stream length 34-113 tokens, mean 70.9 distance of the topic object from the end 27-106, mean 63.9 hello , the status of our reading is low . i wrote again two day s ago . i still waited and there is no result . there is no result with the record number . what is the status of this matter ? ``` The closing question is the same in every stream and carries no information by itself: the answer sits 63.9 tokens behind on average. In about three of every eight streams a second object appears near the end, and it is a trap for any procedure that looks at proximity. This setup is not an exercise but an ordinary production situation. To route a support stream to the right team the topic object is needed, and the newest sentence in the stream almost never carries it. The first path that comes to mind is the cheapest one: look at the last few sentences, throw the rest away. The table below gives the number for that path, and what attention solves is seen only when it is read against that number. #### The Reader That Looks at Distance and the Reader That Does Not Three readers use the same match score and differ only in **which positions they compare**. The fixed window looks at the last k positions; the decaying reader looks at every position but multiplies the weight by 0.94 at every token; attention looks at every position independently of distance. ```python OBJECT_SET = set(OBJECT_NAMES) STATUS_SET = {d for _, ds in OBJECTS.values() for d in ds} def match_score(t): """Match of the query with the position. Learned in a real model; given by hand here.""" return 1.0 if t in OBJECT_SET else (0.25 if t in STATUS_SET else 0.02) def read(seq, window=None, decay=1.0): n = len(seq) start = 0 if window is None else max(0, n - window) best, best_p = None, -1.0 for i in range(start, n): p = match_score(seq[i]) * (decay ** (n - 1 - i)) if p > best_p: best_p, best = p, seq[i] return (best if best in OBJECT_SET else None), n - start BUCKETS = [(0, 32), (33, 64), (65, 999)] def measure(name, **kw): correct, total, compared = [0, 0, 0], [0, 0, 0], 0 for x in DATA: c, m = read(x["seq"], **kw) compared += m j = next(i for i, (a, b) in enumerate(BUCKETS) if a <= x["distance"] <= b) total[j] += 1 correct[j] += c == x["object"] print(f"{name:<26} {compared / STREAMS:>8.1f} {sum(correct) / STREAMS:>7.4f} " + " ".join(f"{d / t:>7.4f}" for d, t in zip(correct, total))) print(f"{'reader':<26} {'compared':>8} {'hit':>7} " + " ".join(f"{a}-{b:<3}".rjust(7) for a, b in BUCKETS)) for k in (8, 16, 32, 64): measure(f"last {k} tokens", window=k) measure("decay with distance", decay=0.94) measure("full attention") print("streams per bucket" + "".join( f"{sum(1 for x in DATA if a <= x['distance'] <= b):>8}" for a, b in BUCKETS).rjust(44)) y = [read(x["seq"], window=16)[0] for x in DATA] print(f"last 16 tokens: {sum(c == DATA[i]['object'] for i, c in enumerate(y))} correct," f" {sum(c is not None and c != DATA[i]['object'] for i, c in enumerate(y))} wrong object," f" {sum(c is None for c in y)} empty response") ``` ``` reader compared hit 0-32 33-64 65-999 last 8 tokens 8.0 0.0000 0.0000 0.0000 0.0000 last 16 tokens 16.0 0.0000 0.0000 0.0000 0.0000 last 32 tokens 32.0 0.0833 1.0000 0.0000 0.0000 last 64 tokens 57.9 0.4917 1.0000 0.9333 0.0000 decay with distance 70.9 0.3333 1.0000 0.5714 0.0000 full attention 70.9 1.0000 1.0000 1.0000 1.0000 streams per bucket 20 105 115 last 16 tokens: 0 correct, 83 wrong object, 157 empty response ``` The triple is read from here. The **bare call** is the cheapest reading: the last 16 tokens, 16 comparisons per stream, hit rate 0.0000. The last line opens up that zero — in 83 streams the second object that enters the window is returned as the wrong answer, in 157 streams an empty response is given because no object is found in the window. **The cheap reading has two separate failure modes, and one is silent, the other is not.** As the window grows the hit rate rises step by step but is always cut at the same place: the last 32 tokens give 1.0000 in the 0-32 bucket and exactly zero beyond it; the last 64 tokens rise to 0.9333 in the 33-64 bucket and again give zero beyond 65 tokens. The window is good where it sees and nothing where it does not. The decaying reader is a more interesting failure. It looks at every position, so its coverage is complete, but it stays at 0.3333. The distance penalty, 0.94 to the power of sixty, drops to about 0.024, and the 1.00 match of the distant object falls below 0.25 of a nearby status root. **Coverage is not enough; the weight must be independent of distance.** This is exactly what attention does, and the 1.0000 hit rate comes from here. #### The Growth Rate of the Cost A weight independent of distance is not free. For a single query, a reading costs n comparisons; but the model does this **for every position**. In a causal pass the i-th position compares the i positions before it, and the total is n(n+1)/2. ```python print(f"{'context n':>9} {'full attention':>14} {'window 64':>12} {'ratio':>7} {'growth':>7}") previous = None for n in (64, 128, 256, 512, 1024): full, win = n * (n + 1) // 2, n * 64 - 64 * 63 // 2 growth = "-" if previous is None else f"{full / previous:.2f}" print(f"{n:>9} {full:>14} {win:>12} {full / win:>7.2f} {growth:>7}") previous = full def sample(seq, seed): r = rng(seed + len(seq)) scores = [match_score(t) for t in seq] u, s = r() * sum(scores), 0.0 for t, w in zip(seq, scores): s += w if u < s: return t if t in OBJECT_SET else None return None print(f"\ngreedy decoding hit {sum(read(x['seq'])[0] == x['object'] for x in DATA) / STREAMS:.4f}" " (same in both runs)") for t in (SEED, SEED + 1): d = sum(sample(x["seq"], t) == x["object"] for x in DATA) / STREAMS print(f"sampling seed {t} hit {d:.4f}") ``` ``` context n full attention window 64 ratio growth 64 2080 2080 1.00 - 128 8256 6176 1.34 3.97 256 32896 14368 2.29 3.98 512 131328 30752 4.27 3.99 1024 524800 63520 8.26 4.00 greedy decoding hit 1.0000 (same in both runs) sampling seed 20260218 hit 0.4292 sampling seed 20260219 hit 0.4042 ``` The growth column is a one-sentence rule: **when the context doubles, the comparisons quadruple.** The fixed window grows linearly, and the ratio between them reaches 8.26 at 1024 tokens. The context window discussion of the next lesson and the whole cost calculation of this curriculum pass between these two curves: why the limit is a number and why it is not enlarged generously is seen from here. The counterpart of this table on the user's side is this: every token added to the context writes a faster-than-linear cost, because it will be compared with all the ones that follow. A document added to the prompt "to be safe" does not pay its cost only with its own length. Latency follows the same curve and is referred to in this curriculum as a **class**: single call, chain, parallel. A single call with a long context can be more expensive than two calls with short contexts, and which to choose is not known without measuring. This lesson measured not what attention is but **what it buys**. Attention does not explain what the model knows; it only determines which positions in the sequence can be linked to each other. Where the match scores come from is still a gap. The last three lines are this lesson's measure of uncertainty. The number of comparisons is a counted quantity and is independent of the run; the reading decision is not. When the largest of the weights is taken, the hit rate is 1.0000 in both runs. When the choice is made from the same weights by **sampling**, the same input gives 0.4292 and 0.4042: two runs, two numbers, the same scaffolding. The selection rule itself is a design decision, and the next lesson measures it. #### Summary - The distinguishing feature of attention is not coverage but **the weight being independent of distance**. The reader with complete coverage but decay with distance stays at 0.3333, while attention gives 1.0000. - Fixed-window reading is good where it sees and nothing where it does not: the last 32 tokens give 1.0000 in the near bucket and 0.0000 beyond it; the last 64 tokens give an overall hit rate of 0.4917. - The cheap reading has two failure modes: in 83 streams the nearby second object is returned as the wrong answer, in 157 streams an empty response comes back. - The cost is the number of comparisons and it grows quadratically: when the context doubles the comparisons quadruple, and at 1024 tokens it is 8.26 times the fixed window. - From the same weights, greedy decoding gives 1.0000 in both runs, while sampling gives 0.4292 and 0.4042. #### Next Step The model can use distant context and the cost of that was counted. But where did this ability come from: the match scores were written by hand in this lesson, and in a real model they are learned. Learning has two separate stages and the two solve separate problems. The next lesson splits a task set in two and counts **how many examples** it takes to improve each: which question is answered from the broad text corpus, which from the field's own examples. ### Pretraining and Fine-Tuning Source: https://academia.sh/en/courses/introduction-to-ai-engineering/pretraining-and-fine-tuning Counting a task set split in two and how many examples each half needs to improve: the simulator pretrained on a fictional general-text corpus of 1,600 sentences, without seeing a single domain example, gives a next-token hit rate of 0.3186 on general text and 0.1983 on domain text; the same measure without pretraining is 0.1296. Fine-tuning with four hundred domain examples raises domain text to 0.5437 but drops general text to 0.2692. The organization's own fault-code mapping is never solved by pretraining: the hit rate stays at 0.0900, below the 0.1200 share of the most frequent code, and the 0.90 threshold is passed only with 400 labeled examples. The previous lesson showed that attention can use distant context and counted its cost, but it wrote the match scores by hand. In a real model those scores are learned. Learning is not a single event: it solves two separate problems, in two separate stages, with two separate data sets. This lesson treats the two stages not as a procedure but as **which problem is solved in which one**. The procedure of transfer learning and pretrained language models was measured in the Architectures topic of the Deep Learning course and in the Natural Language Processing and Computer Vision course; **it is not repeated here**. The question here belongs to the user: there is a task in hand, which stage's product solves it, and if it does not, **how many examples** are needed. The measure is the number of examples. - **LM17.** The general-text corpus and the support tickets are **fictional**: they are generated inside the lesson, seed 20260218. The corpus is 1,600 sentences, the ticket set 600 records. - **LM18.** The model is a **simulator**: pretraining and fine-tuning are **the same procedure** — counting what follows the last one and the last two tokens. The only thing that differs is which text is counted. - **LM19.** **Pretraining** is unlabeled: it sees only text and receives no task label. **Fine-tuning** is done with labeled domain examples and its cost is the number of those examples. - **LM20.** The fine-tuning table is queried **first**, the pretraining table as a **fallback**. If a context has been seen once in fine-tuning, it suppresses the knowledge of pretraining in that context. - **LM21.** The first task is **next-token hit rate** and it is measured on two separate texts: general text and domain text. - **LM22.** The second task is the organization's own mapping: from ticket to **fault code** — a 12-class, fictional and arbitrary code derived from the object and the status. This code **never appears** in general text. - **LM23.** The classifier relies on independent token counts and is built only from fine-tuning examples; at zero examples it has nothing in hand. - **LM24.** The criterion is the same on every row: hit rate on the held-out share (larger is better) and **number of labeled examples** (smaller is better). - **LM25.** The measure of uncertainty is **which ones the fine-tuning examples are**: the same number of examples is selected with a second seed and the second run is printed. #### Two Texts, Two Tasks The general-text corpus consists of ordinary sentences; the tickets use the same grammar but carry the objects and statuses of the domain. The fault code exists only in the tickets. ```python # SIMULATOR — the model is a simulator written in the lesson with the standard # library; no real endpoint is called. The general-text corpus and the support # tickets are FICTIONAL, seed 20260218. SEED, M32, GENERAL, TICKETS = 20260218, 0xFFFFFFFF, 1600, 600 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value SUBJECTS = ["i", "we", "you", "team", "service", "customer", "system", "clerk"] VERBS = ["came", "wrote", "waited", "sent", "opened", "gave", "made", "saw"] NOUNS = ["record", "ticket", "detail", "reply", "result", "number", "address", "process"] ADJECTIVES = ["new", "old", "open", "closed", "normal", "missing"] GENERAL_TEMPLATES = ["{o} {f} a {a} .", "{o} {f} a {b} for {a} .", "this {s} {a} and {o} {f} .", "{o} {f} a {s} {a} .", "{s} {a} but even so {o} {f} ."] CORPUS = [] for i in range(GENERAL): r = rng(SEED + i) CORPUS.append(GENERAL_TEMPLATES[int(r() * 5)].format( o=SUBJECTS[int(r() * 8)], a=NOUNS[int(r() * 8)], b=NOUNS[int(r() * 8)], s=ADJECTIVES[int(r() * 6)], f=VERBS[int(r() * 8)]).split()) OBJECTS = ["invoice", "tariff", "meter", "valve"] STATUSES = ["high", "missing", "broken"] CODES = {(n, d): f"{'ABCD'[i]}{j + 1}" for i, n in enumerate(OBJECTS) for j, d in enumerate(STATUSES)} TICKET_TEMPLATES = ["{o} {f} a record for {n} , state {d} .", "this {n} {d} but even so {o} {f} .", "{o} {f} that {n} status is {d} .", "{n} {d} and {o} {f} the reply ."] DATA = [] for i in range(TICKETS): r = rng(SEED + 100000 + i) n, d = OBJECTS[int(r() * 4)], STATUSES[int(r() * 3)] seq = TICKET_TEMPLATES[int(r() * 4)].format( n=n, d=d, o=SUBJECTS[int(r() * 8)], f=VERBS[int(r() * 8)]).split() DATA.append({"seq": seq, "code": CODES[(n, d)]}) C_TRAIN, C_HOLD = CORPUS[:1400], CORPUS[1400:] TRAIN, HOLD = DATA[:400], DATA[400:] codes = sorted({x["code"] for x in DATA}) print(f"general-text corpus {len(CORPUS)} sentences ({len(C_TRAIN)} training, {len(C_HOLD)} held out)") print(f"support tickets {len(DATA)} ({len(TRAIN)} fine-tuning pool, {len(HOLD)} held out)") print(f"fault code {len(codes)} classes: {' '.join(codes)}") print(f"\ngeneral: {' '.join(C_TRAIN[0])}") print(f"ticket: {' '.join(TRAIN[0]['seq'])} -> {TRAIN[0]['code']}") shared = {t for c in C_TRAIN for t in c} & {t for x in DATA for t in x["seq"]} print(f"corpus vocabulary {len({t for c in C_TRAIN for t in c})}, ticket vocabulary" f" {len({t for x in DATA for t in x['seq']})}, shared {len(shared)}") ``` ``` general-text corpus 1600 sentences (1400 training, 200 held out) support tickets 600 (400 fine-tuning pool, 200 held out) fault code 12 classes: A1 A2 A3 B1 B2 B3 C1 C2 C3 D1 D2 D3 general: old address but even so service saw . ticket: you wrote a record for meter , state missing . -> C2 corpus vocabulary 38, ticket vocabulary 39, shared 27 ``` Of the 39 tokens in the ticket vocabulary, 27 also appear in the general corpus. What separates a ticket is not the grammar but the four objects and three statuses. The fault code, on the other hand, is written in no text: it is the organization's own mapping and can be learned only from the label. This distinction is not a made-up arrangement but the carrier of the measurement. The answer to a task is either somewhere in the text of the world — how a sentence is built, which word follows which — or it lives only in that organization's records. The first comes without asking for any examples, the second makes you pay for every example one by one. The two tables below test the same simulator with the same criterion on two classes of task and write where the difference comes from in terms of the number of examples. #### The Product of Pretraining Pretraining is unlabeled: the simulator only reads the corpus and counts which token follows which. Fine-tuning applies the same procedure to the domain text. The three columns give the same criterion under three separate conditions. ```python HOLD_SEQS = [x["seq"] for x in HOLD] def train(sentences): """Pretraining and fine-tuning are the same procedure: counting what follows the last one and the last two tokens. Only the text differs.""" t = {} for c in sentences: for i in range(1, len(c)): for a in [(c[i - 1],)] + ([(c[i - 2], c[i - 1])] if i > 1 else []): t.setdefault(a, {}) t[a][c[i]] = t[a].get(c[i], 0) + 1 return t def predict(history, tuned, pretrained, default): """Fine-tuning table first, pretraining table as a fallback.""" for t in (tuned, pretrained): for a in (tuple(history[-2:]), tuple(history[-1:])): d = t.get(a) if d: return max(sorted(d), key=d.get) return default def next_hit_rate(sentences, tuned, pretrained, default): correct = total = 0 for c in sentences: for i in range(1, len(c)): total += 1 correct += predict(c[:i], tuned, pretrained, default) == c[i] return correct / total PRETRAINED = train(C_TRAIN) DEFAULT = max(sorted({t for c in C_TRAIN for t in c}), key=lambda t: sum(c.count(t) for c in C_TRAIN)) K_LIST = [0, 2, 4, 8, 16, 32, 64, 128, 256, 400] print(f"{'examples':>8} {'general text':>12} {'domain text':>11} {'untrained domain':>16}") for k in K_LIST: tuned = train([x["seq"] for x in TRAIN[:k]]) if k else {} print(f"{k:>8} {next_hit_rate(C_HOLD, tuned, PRETRAINED, DEFAULT):>12.4f}" f" {next_hit_rate(HOLD_SEQS, tuned, PRETRAINED, DEFAULT):>11.4f}" f" {next_hit_rate(HOLD_SEQS, tuned, {}, DEFAULT):>16.4f}") ``` ``` examples general text domain text untrained domain 0 0.3186 0.1983 0.1296 2 0.2974 0.3305 0.2528 4 0.2992 0.4439 0.3733 8 0.2913 0.4783 0.4102 16 0.2886 0.5360 0.5301 32 0.2727 0.5347 0.5347 64 0.2595 0.5476 0.5476 128 0.2736 0.5502 0.5502 256 0.2736 0.5256 0.5256 400 0.2692 0.5437 0.5437 ``` The first row gives the product of pretraining alone. The simulator that has seen no domain example gets 0.1983 on domain text; the same measure without pretraining is 0.1296. **This is transfer, and it is a number:** the 27 shared tokens make the domain's own text predicted with a clearly higher hit rate. A domain example is not free; pretraining has paid a part of it in advance. The third column becomes equal to the second from 32 examples. When enough domain examples have accumulated, the contribution of pretraining falls to zero: the fallback table is never reached again. **The value of transfer exists while examples are few.** This also says where to look when measuring — in a setup with few examples the difference between a pretrained and an untrained run is worth measuring, in a setup with many examples that difference is zero even if it is measured. The first column, on the other hand, moves in the opposite direction: starting from 0.3186, it falls to 0.2595 at 64 examples, then recovers to 0.2692 at 400. The loss is real and its cause is the order of the tables — a context seen once in fine-tuning suppresses the knowledge of pretraining in that context. While examples are few, this single observation produces an unrepresentative preference; as examples increase the preference approaches the general distribution but does not return to the starting level. **The cost of fine-tuning is not only labeled examples but a loss of 0.0494 in general ability.** #### What Depends on the Label The second task is the organization's own mapping. There is no such thing as a fault code in general text; however much text is read, this mapping cannot be derived. ```python import math def train_classifier(examples): count, total = {}, {} for x in examples: k = x["code"] total[k] = total.get(k, 0) + 1 for t in set(x["seq"]): count[(t, k)] = count.get((t, k), 0) + 1 return count, total def classify(model, seq): count, total = model if not total: return codes[0] best, best_p = None, None for k in sorted(total): p = math.log(total[k]) for t in sorted(set(seq)): p += math.log((count.get((t, k), 0) + 1) / (total[k] + 2)) if best_p is None or p > best_p: best, best_p = k, p return best def pick_sample(pool, k, seed): r, p = rng(seed), list(pool) for i in range(len(p) - 1, 0, -1): j = int(r() * (i + 1)) p[i], p[j] = p[j], p[i] return p[:k] def code_hit_rate(examples): m = train_classifier(examples) return sum(classify(m, x["seq"]) == x["code"] for x in HOLD) / len(HOLD) print(f"{'examples':>8} {'code hit':>12} {'seen classes':>12} {'second run':>13}") threshold = None for k in K_LIST: o = pick_sample(TRAIN, k, SEED) i1, i2 = code_hit_rate(o), code_hit_rate(pick_sample(TRAIN, k, SEED + 1)) print(f"{k:>8} {i1:>12.4f} {len({x['code'] for x in o}):>12} {i2:>13.4f}") if threshold is None and i1 >= 0.90: threshold = k print(f"\nsmallest number of examples passing the 0.90 threshold: {threshold}") print(f"share of the most frequent code in the held-out part:" f" {max(sum(x['code'] == k for x in HOLD) for k in codes) / len(HOLD):.4f}") ``` ``` examples code hit seen classes second run 0 0.0900 0 0.0900 2 0.1200 2 0.0800 4 0.2000 3 0.0950 8 0.2150 6 0.2000 16 0.1600 8 0.2450 32 0.2450 11 0.3600 64 0.4800 12 0.4850 128 0.5700 12 0.6550 256 0.8400 12 0.8350 400 0.9000 12 0.9000 smallest number of examples passing the 0.90 threshold: 400 share of the most frequent code in the held-out part: 0.1200 ``` At zero examples the hit rate is 0.0900 and it is **below** the 0.1200 of the procedure that always says the most frequent code. The contribution of pretraining to this task is not zero but negative: the model knows the language, not the mapping. The seen-classes column gives the mechanical reason — all twelve codes enter the pool only at 64 examples, and a code that has not been seen before cannot be said under any condition. The threshold is 400 examples. Same task, same model, same criterion: **the equivalent of pretraining is zero, that of fine-tuning is 400 examples.** This is the question that splits a task set in two — whether the answer is written in the text or lives only in the organization's records. The curve is not flat either. The hit rate that rises to 0.2150 at eight examples goes down to 0.1600 at sixteen: because the newly arriving classes dilute the existing counts, the number regresses in an intermediate region. Reading an example curve from two points is therefore misleading; to be able to say which number of examples is enough, one has to see where the curve flattens. Here it has not flattened yet: the last 144 examples, from 256 to 400, still add 0.0600 to the hit rate, and the threshold is passed exactly at the last row of the pool. The last column shows the uncertainty. When the number of examples is held fixed and **which examples** are drawn is changed, 128 gives 0.5700 and 0.6550, 32 gives 0.2450 and 0.3600. Both are numbers of the same setup. **An example curve read from a single run hides the variability that comes from the selection of examples.** The counterpart of these two stages on the user's side is not equal. Pretraining is closed inside a component given in this curriculum: it has been done, it cannot be changed, its number can be read only from the surface. Fine-tuning is a decision, and its cost is paid up front — the examples to be labeled, the person who will give the label, the consistency of the label. The table above is the price tag of that decision: 400 examples for 0.90 in this task. The question to ask when deciding is not "can the model do this" but **how many examples it costs to reach this number**. Mechanisms of the later courses, such as prompt design and grounding with retrieval, are ways of approaching the same number **without paying for labels**, and all of them are read against this table. #### Summary - Pretraining and fine-tuning are the same procedure in this simulator; the only thing that differs is which text is counted. The texts and the fault code are fictional, seed 20260218. - The contribution of pretraining to domain text is measurable: 0.1983 without seeing a single domain example, 0.1296 without pretraining. The contribution exists while examples are few and falls to zero from 32 examples. - The cost of fine-tuning is not only 400 labeled examples: the general-text hit rate falls from 0.3186 to 0.2692, and the lowest point is at 64 examples, at 0.2595. - The organization's own mapping is never solved by pretraining: a hit rate of 0.0900 is below the 0.1200 share of the most frequent code. The 0.90 threshold is passed only with 400 labeled examples. - When the same number of examples is selected with another seed, the number moves: at 128 examples, 0.5700 and 0.6550. #### Next Step Where the model learns what from was split, and the cost of each stage was written in numbers of examples. But in this lesson the model always produced a single prediction: the most likely token, in a single step. A real response is not a single token; it is produced step by step and every step is a choice. The next lesson measures that choice: the output length against the number of tokens produced, greedy decoding against sampling in two runs on the same prompt, and where the distribution temperature stands between the two. ### Inference Source: https://academia.sh/en/courses/introduction-to-ai-engineering/inference The cost of producing a response step by step and a count for the selection rule: in 200 fictional tickets greedy decoding takes 6.43 tokens and 211.1 comparisons per response with a hit rate of 1.0000. When the maximum output length is pulled down to 4, all responses hit the limit and the hit rate drops to 0.0000, to 0.5450 at 6, to 0.8100 at 7. On the same prompt, as the distribution temperature of sampling rises from 0.2 to 2.0 the hit rate falls from 0.9900 to 0.5600 and the share of responses that change between two runs rises from 0.5550 to 0.8600; with greedy decoding this share is 0.0000. With the input fixed at 30 tokens, when the output grows to 512 tokens the output itself accounts for 0.8953 of the comparisons. The previous lesson separated where the model learns what from, but it produced a single prediction everywhere: the most likely token, in a single step. A real response is not a single token. The model produces a token, appends it to the end of its own input, produces the next one, and keeps going until the stop condition arrives. What we call **inference** is this loop; both its cost and its uncertainty are born here. Two questions are measured. The first is cost: how much work is done as a response gets longer. The second is the selection rule: at each step, is the largest of the distribution taken, or is the choice **sampled** from the distribution — and what does this decision give in two runs on the same prompt. This lesson directly counts the third claim of the curriculum: the model is an uncertain component and the amount of uncertainty is a setting. - **LM26.** The tickets and the response samples are **fictional**: they are generated inside the lesson, seed 20260218. The sample set is 600 responses, the test set 200 tickets. - **LM27.** The model is a **simulator**: at every step it builds a distribution for the condition and the last two tokens, chooses a token and appends its choice to the history. - **LM28.** The **stop condition** is the period. The **maximum output length** is a separate limit and when it is exceeded the response is cut midway. - **LM29.** 0.15 of the response samples are a general pattern unrelated to the domain. For this reason the model does not have a single correct path under every condition. - **LM30.** **Hit rate** is counted only for completed responses: the response must end with a period and carry the correct object and the correct action. A response cut midway counts as wrong. - **LM31.** **Cost** is written in two units: the number of tokens produced and the number of comparisons. Each step costs as many comparisons as the input length plus the tokens produced up to that point. - **LM32.** The input length is 30 tokens in all measurements; the only thing that changes is the output. - **LM33.** **Distribution temperature** raises the probabilities to a power: zero is greedy decoding, and as it grows the distribution flattens. - **LM34.** The measure of uncertainty is **two runs of the same prompt**: the second run is taken by changing the seed and the share of responses that come out different in the two runs is printed. #### Producing a Response The simulator does not write a response at once. It starts with the start marker, builds a distribution at each step, chooses a token and appends its choice to the end of the history. ```python # SIMULATOR — the model is a simulator written in the lesson with the standard # library; no real endpoint is called. Tickets and response samples are FICTIONAL, seed 20260218. SEED, M32, SAMPLES, TEST = 20260218, 0xFFFFFFFF, 600, 200 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value OBJECTS = ["invoice", "tariff", "meter", "valve"] STATUSES = ["high", "missing", "broken"] ACTIONS = {"high": "objection record opened", "missing": "information sent", "broken": "field team dispatched"} TEMPLATES = ["{n} for {e} .", "{n} record examined and {e} .", "{n} condition for {e} ."] GENERIC = ["issue examined and result sent .", "request received and issue opened ."] def make_samples(seed, count): v = [] for i in range(count): r = rng(seed + i) n, d = OBJECTS[int(r() * 4)], STATUSES[int(r() * 3)] if r() < 0.15: y = GENERIC[int(r() * 2)] else: y = TEMPLATES[int(r() * 3)].format(n=n, e=ACTIONS[d]) v.append({"condition": (n, d), "response": [""] + y.split()}) return v TRAIN_SET = make_samples(SEED, SAMPLES) TEST_SET = make_samples(SEED + 500000, TEST) TABLE = {} for x in TRAIN_SET: y = x["response"] for i in range(1, len(y)): for a in ((x["condition"], tuple(y[max(0, i - 2):i])), (x["condition"], (y[i - 1],))): TABLE.setdefault(a, {}) TABLE[a][y[i]] = TABLE[a].get(y[i], 0) + 1 def distribution(condition, history): d = TABLE.get((condition, tuple(history[-2:]))) or TABLE.get( (condition, (history[-1],))) or {".": 1} t = sum(d.values()) return sorted((b, s / t) for b, s in d.items()) def generate(condition, input_len, temperature=0.0, seed=SEED, max_output=12): """Step-by-step generation: at each step a distribution is built, a token is chosen and appended to the history. The stop condition is the period, the upper limit is max_output tokens.""" r, history, compared = rng(seed + input_len), [""], 0 while len(history) - 1 < max_output: d = distribution(condition, history) compared += input_len + len(history) - 1 if temperature <= 0: choice = max(d, key=lambda bp: (bp[1], bp[0]))[0] else: a = [(b, p ** (1 / temperature)) for b, p in d] u, s = r() * sum(p for _, p in a), 0.0 choice = a[-1][0] for b, p in a: s += p if u < s: choice = b break history.append(choice) if choice == ".": return history[1:], compared, "condition" return history[1:], compared, "limit" x = TEST_SET[0] print(f"fictional samples {SAMPLES} responses, test {TEST} tickets") print(f"ticket condition {x['condition']}, correct action: {ACTIONS[x['condition'][1]]}") for name, temp, sd in (("greedy decoding", 0.0, SEED), ("greedy decoding", 0.0, SEED + 1), ("sampling 1.0", 1.0, SEED), ("sampling 1.0", 1.0, SEED + 1)): y, k, stop = generate(x["condition"], 30, temp, sd, 12) print(f"{name:<16} seed {sd} {len(y):>2} tokens {stop:<9} {' '.join(y)}") ``` ``` fictional samples 600 responses, test 200 tickets ticket condition ('meter', 'high'), correct action: objection record opened greedy decoding seed 20260218 8 tokens condition meter record examined and objection record opened . greedy decoding seed 20260219 8 tokens condition meter record examined and objection record opened . sampling 1.0 seed 20260218 8 tokens condition meter record examined and objection record opened . sampling 1.0 seed 20260219 7 tokens condition meter condition for objection record opened . ``` The four lines say two things. Greedy decoding gives the **exactly same** response with both seeds: the seed enters nowhere, because no draw is made at any step. Sampling, on the other hand, gives with the second seed another sentence of 7 tokens instead of 8. Both are correct; **changing and being wrong are not the same thing**, and this distinction is what the rest of the lesson measures. Another consequence of the loop shows in latency. The first token of the response is ready after a single step, the last one only after m steps. For this reason the latency of a language model call is not a single number: the time to the first token and the time to the completion of the response behave separately, and the second grows with the output length. This curriculum does not write time in a raw unit but keeps the distinction; a response of eight tokens and a response of eighty tokens are not in the same latency class. Step-by-step selection also has a structural limit: each step chooses the best for itself and does not look ahead. In the response above, the moment the second token `record` is chosen, the sentence is bound to the long pattern; had `for` been chosen at that step, the sentence would have ended two tokens shorter. **Greedy decoding is the most likely per step, not per sentence.** This means the output length cannot be controlled exactly either, and the limit problem measured by the next section is born from here. #### The Cost of Step-by-Step Generation Each step rereads all of those before it. If a response is m tokens long and the input is n tokens, the total number of comparisons is m·n plus m(m+1)/2: the first term is the repeated reading of the input, the second is the output's own tail. ```python INPUT_LEN = 30 def is_correct(y, condition): """A response counts only if it is complete and carries the correct object and the correct action.""" return y[-1] == "." and condition[0] in y and ACTIONS[condition[1]].split()[0] in y def run(temperature, seed, max_output): tokens = compared = hits = limited = 0 responses = [] for i, x in enumerate(TEST_SET): y, k, stop = generate(x["condition"], INPUT_LEN, temperature, seed + i, max_output) tokens += len(y) compared += k hits += is_correct(y, x["condition"]) limited += stop == "limit" responses.append(tuple(y)) n = len(TEST_SET) return tokens / n, compared / n, hits / n, limited / n, responses print(f"{'max output':>15} {'tokens':>9} {'comparisons':>14}" f" {'at the limit':>15} {'hit rate':>8}") for mo in (4, 6, 7, 8, 12): b, k, i, s, _ = run(0.0, SEED, mo) print(f"{mo:>15} {b:>9.2f} {k:>14.1f} {s:>15.4f} {i:>8.4f}") print(f"\n{'output m':>8} {'comparisons':>14} {'output share':>14}") for m in (8, 32, 128, 512): k = m * INPUT_LEN + m * (m + 1) // 2 print(f"{m:>8} {k:>14} {m * (m + 1) // 2 / k:>14.4f}") ``` ``` max output tokens comparisons at the limit hit rate 4 4.00 126.0 1.0000 0.0000 6 5.79 187.7 0.4550 0.5450 7 6.25 204.0 0.1900 0.8100 8 6.43 211.1 0.0000 1.0000 12 6.43 211.1 0.0000 1.0000 output m comparisons output share 8 276 0.1304 32 1488 0.3548 128 12096 0.6825 512 146688 0.8953 ``` The triple is read from here. The **bare call** is the narrowest setting: maximum output 4 tokens, 126 comparisons per response, hit rate **0.0000**. It is cheap and useless, because all of the responses hit the limit and are cut midway. The **scaffolding** is opening the limit to the real length of the response: at 8 tokens none hits the limit and the hit rate becomes 1.0000. The **cost** is going from 4.00 to 6.43 tokens and from 126.0 to 211.1 comparisons — about 1.68 times. The intermediate rows show what kind of failure the limit produces. At six tokens 0.4550 of the responses are cut and the hit rate stays at 0.5450; at seven the share is 0.1900 and the hit rate 0.8100. **This failure is silent:** a cut response returns no error, it looks like the beginning of a proper sentence. The share hitting the limit is a separate number that has to be measured. Going from eight to twelve changes nothing: 6.43 and 211.1 stay the same. The maximum output length is a **ceiling**, not a target; when the stop condition comes earlier, the size of the ceiling creates no cost. This also says how to choose the limit: keeping the ceiling narrow brings no saving, it only creates a risk of being cut. The place to save is the length of the prompt and of the expected response, not the ceiling. The second table gives where the cost slides in long outputs. With the input fixed at 30 tokens, when the output is 8 tokens the output accounts for only 0.1304 of the comparisons; at 512 tokens this share rises to 0.8953. **The cost of a long response depends not on the length of the prompt but on its own length.** #### The Selection Rule and Temperature At each step the largest of the distribution can be taken, or the choice can be sampled from the distribution. Temperature is the continuous setting between the two: a small temperature sharpens the distribution, a large temperature flattens it. Zero temperature is the limiting case of sampling and the draw disappears entirely. The sweep below calls the same 200 tickets at seven settings, two runs each; the only thing that changes is the selection rule, and the model and the prompt are the same on every row. ```python print(f"{'temperature':>11} {'hit 1':>9} {'hit 2':>9} {'changed':>8}" f" {'tokens':>9} {'comparisons':>14}") for temp in (0.0, 0.2, 0.5, 0.8, 1.0, 1.5, 2.0): b, k, i1, _, y1 = run(temp, SEED, 16) _, _, i2, _, y2 = run(temp, SEED + 1000, 16) d = sum(a != b2 for a, b2 in zip(y1, y2)) / len(y1) label = f"{temp:.1f}" print(f"{label:>11} {i1:>9.4f} {i2:>9.4f} {d:>8.4f} {b:>9.2f} {k:>14.1f}") ``` ``` temperature hit 1 hit 2 changed tokens comparisons 0.0 1.0000 1.0000 0.0000 6.43 211.1 0.2 0.9900 0.9900 0.5550 6.47 212.5 0.5 0.9500 0.9400 0.6650 6.53 214.3 0.8 0.8800 0.8750 0.7200 6.50 213.5 1.0 0.7900 0.8200 0.7650 6.47 212.2 1.5 0.6250 0.6700 0.8550 6.47 212.1 2.0 0.5600 0.5600 0.8600 6.46 211.5 ``` The greedy decoding row is the plainest measure of uncertainty in this curriculum: the share of responses that change between two runs is **0.0000**. The model is not uncertain here; the uncertainty comes from the selection rule, not from the model itself. The second row carries the real warning. At temperature 0.2 the hit rate is almost undamaged at 0.9900, but **0.5550 of the responses change** between two runs. The same ticket gets two different sentences at two different times, and both are correct. For a product this number is independent design data: if the text of the response is stored somewhere, compared, or matched against an expected output in a test, 0.5550 is a test that breaks on every run. As the temperature grows the two columns separate. The change between two runs saturates toward 0.8600, while the hit rate falls from 0.9900 to 0.5600. **The cost of the scaffolding is paid here in hit rate, and what is received in return is only diversity.** The token and comparison columns are almost constant: the cost of temperature is not tokens but accuracy. It can also be read from the table that three temperatures serve three separate jobs. In a classification or code-assignment task 0.0 is the right setting: repeatable and with the highest hit rate. In a response text, between 0.2 and 0.5 lowers the hit rate very little and takes the text out of monotony. Above 1.0 can be defended only if diversity itself is the purpose of the product, and in this table it costs a hit-rate loss of 0.2100. The table has one more column that stays silent: the cost of a wrong response. At temperature 1.0, 0.2100 of the responses are wrong, and because these errors are randomly distributed they can be corrected by calling the same prompt again. But calling again is not free — every retry is a new 211.1 comparisons and 6.47 tokens. In a scaffolding where one of five tickets is called a second time, the average cost rises to 1.21 calls, and the hit rate obtained in return is below the already 1.0000 figure of temperature 0.0. **Opening up the uncertainty first and then closing it with a retry is a more expensive way of reaching the same number.** This is the form the second claim of the curriculum takes in this lesson: the cost of the added scaffolding is certain, its gain is uncertain. #### Summary - Inference is not a single operation but a loop: each step builds a distribution, chooses a token and appends its choice to the end of the input. The tickets and the response samples are fictional, seed 20260218. - Cost is written in two units. With greedy decoding, 6.43 tokens and 211.1 comparisons per response; with the input fixed at 30 tokens, when the output grows to 512 tokens the output itself accounts for 0.8953 of the comparisons. - The maximum output length is a source of silent failure: at 4 tokens all responses are cut and the hit rate is 0.0000, at 6 it is 0.5450, at 7 0.8100, at 8 1.0000. - Greedy decoding gives exactly the same response in two runs (share changed 0.0000). The uncertainty comes from the selection rule, not from the model. - At temperature 0.2 the hit rate stays at 0.9900 while the share of responses that change between two runs is 0.5550; at 2.0 the hit rate is 0.5600 and the share changed is 0.8600. The token cost is almost constant throughout this sweep. #### Next Step How the response is produced and what the selection rule costs was counted. But in all of these measurements the input was held fixed at 30 tokens. In a real scaffolding the input is not fixed: the ticket stream grows, documents are added, earlier responses are put back. The model also has a **window**, and that window is fixed. The next lesson measures what happens when the window is exceeded: how many tokens dropping from the head, dropping from the middle and summarizing save on the same task, and how much they lower the hit rate. ### Context Window Source: https://academia.sh/en/courses/introduction-to-ai-engineering/context-window Measuring what happens when the window is exceeded and how many tokens three truncation decisions save and how much they lower the hit rate: in a window of 256 tokens, 0.5967 of 300 fictional prompts do not fit, and the bare call gives a full-response rate of 0.3833 at 282.3 tokens. Dropping from the head gives 0.7433 at 217.5 tokens, dropping from the middle gives 0.7367 at the same tokens, and summarizing raises the code hit rate to 0.9667 at 100.6 tokens but drops the step hit rate to 0.0000. Which decision truncation makes depends on the position of the evidence: dropping from the head gives 0.5351 in the head bucket, dropping from the middle gives 0.3627 in the middle bucket. When the window goes from 128 to 384, the full response rises from 0.3767 to 0.9867. The previous lesson counted how a response is produced step by step and what the selection rule costs, but it kept the input fixed at 30 tokens. In a real scaffolding the input is not fixed. The prompt grows: an instruction, a few of the field team's documents, the ticket stream and the question are put side by side. The model, on the other hand, has a fixed **context window**, and when that window is exceeded something happens. This lesson measures that "something". The window is a limit; what will truncate a prompt that exceeds the limit is a **decision**, and the decision belongs to the engineer. Three decisions are put side by side on the same task — dropping from the head, dropping from the middle, summarizing — and each is written with two numbers: how many tokens it saved, how much it lowered the hit rate. - **LM35.** The document pool and the tickets are **fictional**: they are generated inside the lesson, seed 20260218. The pool is 24 documents, each 28 tokens; the records come from no source system. - **LM36.** A document carries a header line (document number, object, status, fault code) and five procedure steps. - **LM37.** The prompt has four parts: instruction, documents, ticket, question. Truncation is done at **document integrity**; no half document is left. The instruction, the ticket and the question are never dropped in any decision. - **LM38.** The task has two fields: the **fault code** and the **second step** of the correct document. A full response is both being correct at once. - **LM39.** The model is a **simulator**. It reads the prompt as long as it fits in the window; if it does not fit, the call fails with a **window error** and no response returns. - **LM40.** The second failure of the simulator is the **timeout**, and its share grows with the prompt length. This share depends on the seed; the second run is taken by changing the seed. - **LM41.** The summarizing procedure is done here by a rule: only the header line of each document is left. In a real scaffolding the summary is a model call and has **a cost of its own**. - **LM42.** The criterion is the same on every row: tokens per prompt (smaller is better), share of failure (smaller is better), code and step hit rates and full-response rate (larger is better). - **LM43.** The **position** of the evidence in the prompt is recorded: the order of the target document in the document list. - **LM44.** In the window sweep the share of failure is not counted; that table is independent of the run. - **LM45.** The **bare call** in this lesson is sending the prompt without any truncation. #### A Prompt That Exceeds the Window The prompt starts with the instruction, continues with four to fourteen documents, and ends with the ticket and the question. The window is 256 tokens. ```python # SIMULATOR — the model is a simulator written in the lesson with the standard # library; no real endpoint is called. The document pool and the tickets are FICTIONAL, seed 20260218. SEED, M32, WINDOW, TICKETS = 20260218, 0xFFFFFFFF, 256, 300 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value OBJECTS = ["invoice", "tariff", "meter", "valve", "seal", "line"] STATUSES = ["high", "missing", "broken", "old"] ACTIONS = ["open record", "dispatch team", "repeat reading", "send information", "compare tariff", "replace seal"] INSTRUCTION = ("instruction : read the documents below , find the one matching " "this ticket , write the code and second step .").split() QUESTION = "question : what is the code and second step ?".split() DOCUMENT_POOL = [] for i, n in enumerate(OBJECTS): for j, d in enumerate(STATUSES): r = rng(SEED + 900 + i * 4 + j) b = ["document", str(101 + i * 4 + j), ":", n, d, "code", f"{'ABCDEF'[i]}{j + 1}"] for a in range(1, 6): b += ["step", str(a)] + ACTIONS[int(r() * 6)].split() DOCUMENT_POOL.append({"seq": b + ["end"], "object": n, "status": d, "code": b[6], "step2": b[13:15]}) DATA = [] for i in range(TICKETS): r = rng(SEED + i) h = int(r() * 24) target = DOCUMENT_POOL[h] others = [DOCUMENT_POOL[(h + 1 + int(r() * 23)) % 24] for _ in range(3 + int(r() * 11))] position = int(r() * (len(others) + 1)) documents = ([b["seq"] for b in others[:position]] + [target["seq"]] + [b["seq"] for b in others[position:]]) tail = ["ticket", ":", target["object"], "status", target["status"], ",", "reply", "awaited", "."] + QUESTION DATA.append({"documents": documents, "tail": tail, "position": position, "code": target["code"], "step2": target["step2"], "object": target["object"], "status": target["status"]}) def join_prompt(documents, tail): d = list(INSTRUCTION) for b in documents: d += b return d + tail lengths = [len(join_prompt(x["documents"], x["tail"])) for x in DATA] print(f"fictional document pool {len(DOCUMENT_POOL)} documents, each" f" {len(DOCUMENT_POOL[0]['seq'])} tokens") print(f"{TICKETS} prompts: {min(lengths)}-{max(lengths)} tokens, mean {sum(lengths) / TICKETS:.1f};" f" document count {min(len(x['documents']) for x in DATA)}" f"-{max(len(x['documents']) for x in DATA)}") print(f"window {WINDOW} tokens; prompts exceeding the window" f" {sum(u > WINDOW for u in lengths) / TICKETS:.4f}") print("\n" + " ".join(DOCUMENT_POOL[9]["seq"])) print(" ".join(DATA[0]["tail"])) ``` ``` fictional document pool 24 documents, each 28 tokens 300 prompts: 152-432 tokens, mean 282.3; document count 4-14 window 256 tokens; prompts exceeding the window 0.5967 document 110 : meter missing code C2 step 1 send information step 2 send information step 3 open record step 4 replace seal step 5 compare tariff end ticket : line status old , reply awaited . question : what is the code and second step ? ``` The mean prompt is 282.3 tokens and the window is 256: **0.5967 of the prompts do not fit.** This share is a property of the scaffolding, not of the model. As the number of documents goes from 4 to 14 the prompt grows from 152 tokens to 432, and which side of the limit one stays on is decided by the procedure that builds the prompt. #### Three Truncation Decisions The decisions work at document integrity. Dropping from the head removes the oldest document, dropping from the middle removes the one in the middle of the list; summarizing drops no document but leaves only the header of each. A fourth decision combines the two: summarize everything, then put the last documents back in full as far as the budget allows. ```python def drop_head(x, p): bl = list(x["documents"]) while len(bl) > 1 and len(join_prompt(bl, x["tail"])) > p: bl.pop(0) return join_prompt(bl, x["tail"]) def drop_middle(x, p): bl = list(x["documents"]) while len(bl) > 1 and len(join_prompt(bl, x["tail"])) > p: bl.pop(len(bl) // 2) return join_prompt(bl, x["tail"]) def summarize(x, p): return join_prompt([b[:7] for b in x["documents"]], x["tail"]) def summarize_keep_tail(x, p): bl = [b[:7] for b in x["documents"]] for i in range(len(bl) - 1, -1, -1): attempt = bl[:i] + [x["documents"][i]] + bl[i + 1:] if len(join_prompt(attempt, x["tail"])) > p: break bl = attempt return join_prompt(bl, x["tail"]) def answer(seq, object_name, status): code = step = None for i in range(len(seq) - 6): if (seq[i] == "document" and seq[i + 3] == object_name and seq[i + 4] == status and seq[i + 5] == "code"): code = seq[i + 6] for j in range(i + 7, len(seq) - 3): if seq[j] == "end": break if seq[j] == "step" and seq[j + 1] == "2": step = seq[j + 2:j + 4] break break return code, step def call(prompt, seed, i): """Failures of the simulator: the window overflow is certain, the timeout is a share that grows with the prompt length.""" if len(prompt) > WINDOW: return "window" return "timeout" if rng(seed + i)() < 0.02 + 0.06 * len(prompt) / WINDOW else None def measure(name, trim, seed=SEED, write=True): tokens = window_errors = timeouts = code_hits = step_hits = full = 0 for i, x in enumerate(DATA): prompt = trim(x, WINDOW) if trim else join_prompt(x["documents"], x["tail"]) tokens += len(prompt) h = call(prompt, seed, i) window_errors += h == "window" timeouts += h == "timeout" if h: continue k, a = answer(prompt, x["object"], x["status"]) code_hits += k == x["code"] step_hits += a == x["step2"] full += k == x["code"] and a == x["step2"] n = len(DATA) if write: print(f"{name:<22} {tokens / n:>8.1f} {window_errors / n:>8.4f} {timeouts / n:>7.4f}" f" {code_hits / n:>7.4f} {step_hits / n:>7.4f} {full / n:>7.4f}") return full / n print(f"{'decision':<22} {'tokens':>8} {'window':>8} {'timeout':>7}" f" {'code':>7} {'step':>7} {'full':>7}") DECISIONS = [("bare call", None), ("drop head", drop_head), ("drop middle", drop_middle), ("summarize", summarize), ("summarize, keep tail", summarize_keep_tail)] for name, f in DECISIONS: measure(name, f) ``` ``` decision tokens window timeout code step full bare call 282.3 0.5967 0.0200 0.3833 0.3833 0.3833 drop head 217.5 0.0000 0.0600 0.7433 0.7433 0.7433 drop middle 217.5 0.0000 0.0600 0.7367 0.7367 0.7367 summarize 100.6 0.0000 0.0333 0.9667 0.0000 0.0000 summarize, keep tail 221.9 0.0000 0.0633 0.9367 0.7167 0.7167 ``` The **bare call** spends 282.3 tokens and the full-response rate is 0.3833. The source of the loss is not the hit rate but **failure**: 0.5967 of the calls give no response at all because they do not fit in the window. A call that hits the limit does not produce a wrong answer, it produces no answer. **Dropping from the head** is the plainest truncation: it brings the tokens down to 217.5, zeroes the window error and raises the full response to 0.7433. The **cost** has three items — the dropped documents, an extra truncation step and the timeout share rising from 0.0200 to 0.0600. This last item goes against intuition: truncation did not shorten the average prompt, it **lengthened** it, because long prompts that previously could not be sent at all are now sent. **Summarizing** is the cheapest decision: 100.6 tokens, 0.3564 of the bare call. It raises the code hit rate to 0.9667 — because all the document headers stay, the correct code is always found. The step hit rate, however, is **0.0000** and the full-response rate is zero too. The summary has thrown away the thing that was asked. **How much a summary loses is read not from the summary itself but from the question.** The fourth decision gives two separate answers on two criteria: on code hit rate it passes dropping from the head by a wide margin, 0.9367 against its 0.7433, on full response it stays below it at 0.7167 and is 4.4 tokens more expensive. **Which one wins depends on which column is read**, and the choice of column is a product decision. #### Where Truncation Cuts The hit rate of truncation does not fit in a single number: whether the document a decision drops is accurate or destructive depends on where the sought evidence stands. ```python DECISIONS = [("drop head", drop_head), ("drop middle", drop_middle), ("summarize", summarize), ("summarize, keep tail", summarize_keep_tail)] def bucket(x): n = len(x["documents"]) return 0 if x["position"] < n / 3 else (1 if x["position"] < 2 * n / 3 else 2) print(f"{'decision':<22} {'head':>7} {'middle':>7} {'tail':>7} {'run 2':>9}") for name, f in DECISIONS: d, t = [0, 0, 0], [0, 0, 0] for i, x in enumerate(DATA): prompt = f(x, WINDOW) j = bucket(x) t[j] += 1 if call(prompt, SEED, i): continue k, a = answer(prompt, x["object"], x["status"]) d[j] += k == x["code"] and a == x["step2"] second = measure(name, f, SEED + 7000, False) print(f"{name:<22} " + " ".join(f"{a / b:>7.4f}" for a, b in zip(d, t)) + f" {second:>9.4f}") print("tickets per bucket" + "".join(f"{t[j]:>8}" for j in range(3)).rjust(28)) print(f"\n{'window':>8} {'drop head':>11} {'summarize keep':>15} {'summarize':>9}") for p in (128, 192, 256, 384): s = [] for f in (drop_head, summarize_keep_tail, summarize): hit = 0 for i, x in enumerate(DATA): prompt = f(x, p) if len(prompt) > p: continue k, a = answer(prompt, x["object"], x["status"]) hit += k == x["code"] and a == x["step2"] s.append(hit / len(DATA)) print(f"{p:>8} {s[0]:>11.4f} {s[1]:>15.4f} {s[2]:>9.4f}") ``` ``` decision head middle tail run 2 drop head 0.5351 0.8235 0.9286 0.7367 drop middle 0.9474 0.3627 0.9048 0.7000 summarize 0.0000 0.0000 0.0000 0.0000 summarize, keep tail 0.5439 0.7353 0.9286 0.7000 tickets per bucket 114 102 84 window drop head summarize keep summarize 128 0.3767 0.2400 0.0000 192 0.6367 0.6100 0.0000 256 0.8000 0.7667 0.0000 384 0.9867 0.9833 0.0000 ``` The first three columns open up the totals of the decisions, which are close to each other. Dropping from the head gives 0.5351 in the head bucket and 0.9286 in the tail bucket; dropping from the middle rises to 0.9474 in the head bucket but falls to 0.3627 in the middle bucket. **The total hit rates of the two decisions are almost the same (0.7433 and 0.7367) and their error distributions are each other's opposite.** For a product these are not the same decision: if the evidence sits in the old records, dropping from the head errs systematically, if it sits in the middle of the stream, dropping from the middle does. The last column is the second run of the same scaffolding. Only the timeout draw changed, and dropping from the head gave 0.7367 instead of 0.7433, dropping from the middle 0.7000 instead of 0.7367. **The same truncation decision, the same data, two runs, two numbers.** The difference of 0.0066 between the decisions falls below this variability: this table read from a single run would have made dropping from the head the winner, and the second run makes the two equal. The lower table leaves the share of failure out and is independent of the run. As the window goes from 128 to 384 the full response of dropping from the head rises from 0.3767 to 0.9867: the whole of the truncation decision is the effort to close the shortfall in the window, and if the window is big enough no decision is needed. The column of summarizing is 0.0000 at every window — **a loss that does not come from the window is not closed by enlarging the window.** As long as the lost information stays outside the prompt it does not come back whatever the limit is, and the question that separates these two tables is this: does the loss come from the limit, or from the decision made to meet the limit. #### Summary - The context window is a fixed limit; when it is exceeded this simulator gives no response at all. In the bare call 0.5967 of the prompts do not fit and the full-response rate stays at 0.3833. The documents and the tickets are fictional, seed 20260218. - Dropping from the head brings the prompt from 282.3 down to 217.5 tokens and raises the full response to 0.7433. Its cost is the dropped documents, an extra step and the timeout share rising from 0.0200 to 0.0600. - Summarizing is the cheapest decision (100.6 tokens) and raises the code hit rate to 0.9667, but because it throws away the step that was asked, the full-response rate is 0.0000. - The hit rate of truncation depends on the position of the evidence: dropping from the head gives 0.5351 in the head bucket, dropping from the middle gives 0.3627 in the middle bucket. Their totals are almost the same, their error distributions are opposite. - The same scaffolding gives 0.7367 instead of 0.7433 in the second run; the difference between the decisions is below this variability. When the window goes from 128 to 384 the full response rises from 0.3767 to 0.9867. #### Next Step In five lessons the surface of the model was measured: which unit it works with, why it can use distant context, what it learned from where, how it produces a response and where its limit lies. The window is a limit and how to fill it is a decision — this lesson compared three forms of that decision with numbers. But until now everything has been done under the same assumption: that if the correct information is put into the prompt, the model will use it correctly. **What the model does well** was never asked. The next lesson splits a task set into task classes and counts in which class the bare call collapses. ### Capabilities and Limits Source: https://academia.sh/en/courses/introduction-to-ai-engineering/capabilities-and-limits Measuring five task classes with the bare call on twenty-four fictional support requests: summarization 0.7500, classification 0.8750, ranking 0.6250, counting 0.4167 and exact arithmetic 0.2917. When the seed is changed the number of the first three classes moves, the last two do not move at all, and the number of requests whose response changed comes out 0/24. The counting hit rate equals exactly the 10/24 requests that carry no inflected form, the exact-arithmetic hit rate equals exactly the 7/24 requests that carry no carry across a piece boundary. The three-run majority scaffolding changes nothing in these two classes with three times the tokens, the procedure scaffolding that takes the calculation out of the tokens carries both to 1.0000. The previous lesson measured the context window as a limit and showed that how to fill the window is a decision. The question of how many tokens the truncation decisions save and how much they lower the hit rate was answered. What was not asked comes earlier: whatever is put into the window, **what the model does well** was never measured. If a task is not going well, is the fault in the prompt, in the window, or in the task itself? This lesson runs five task classes on the same data with the same bare call and writes the hit rate per class. In two classes the number collapses, and the source of the collapse is not in the prompt, it is **at the token level**. - **LM46. The fictional request set is 24 short Turkish support requests.** The requests are generated from the combination of five main sentences and six extra sentences, and two meter readings stand at the end of each request. The texts, the classes and the readings are **fictional**. - **LM47. The model is a simulator**; no real endpoint is called. The simulator is written with the standard library, its seed is **20260218**, and it produces its response only from the token sequence. - **LM48. The tokenizer is fictional** and carries on the measure set up in the first lesson of this topic: letter pieces are split into fours, digit pieces into twos. - **LM49. Five task classes are measured:** summarization, classification, ranking, counting and exact arithmetic. The measure of each class is the **hit rate**: in how many of the twenty-four requests the response is correct. - **LM50. The cost is written in three units:** input tokens per request, number of calls and latency class. Raw time is not written. - **LM51. The uncertainty is measured with two runs.** The same task set is run with seed 20260218 and seed 20260219; in addition, the **number of requests whose response changed** between the two runs is printed. #### Bare Call on Five Task Classes ```python # COMMON -- FICTIONAL. The simulator below does not call a real endpoint; it is written with the # standard library and carries the measurable surface of a language model. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def tokenize(text): """Fictional tokenizer: letter pieces are split into fours, digit pieces into twos.""" out, buffer = [], "" for ch in text + " ": if ch.isalnum(): buffer += ch continue if buffer: n = 2 if buffer.isdigit() else 4 out += [buffer[i:i + n] for i in range(0, len(buffer), n)] buffer = "" if ch.strip(): out.append(ch) return out MAIN = [ # (main sentence, true class) ("sayac okunmadi tahmini fatura cikti", "olcum"), ("fatura tutari gecen aya gore iki kat cikti", "odeme"), ("gece boyunca su basinci dustu", "tesisat"), ("planli kesinti bildirimi ulasmadi", "kesinti"), ("abonelik devri sayaca islenmedi", "sozlesme"), ] EXTRA = ["sayacin numarasi degistirilmis olabilir", "adres bilgisi eski gorunuyor", "onceki ay ayni sorun yasanmadi", "sayaca ulasilamiyor kapi kapali", "kesinti suresi beklenenden uzun surdu", "otomatik talimat sayacta yok"] LEXICON = { # class -> pieces the simulator recognizes and fictional priority "kesinti": ({"kesi", "bild"}, 1), "olcum": ({"okun", "olcu", "tahm"}, 2), "tesisat": ({"basi", "vana", "tesi", "dust"}, 3), "odeme": ({"fatu", "odem", "tuta", "tali"}, 4), "sozlesme": ({"abon", "devr", "sozl", "adre"}, 5), } CLASSES = list(LEXICON) TASKS = {"summarization": 11, "classification": 23, "ranking": 41, "counting": 59, "exact_arithmetic": 71} SAMPLES = {} REQUESTS = [] for i in range(24): r = rng(SEED + 31 * i) a, b = 100000 + int(r() * 380000), 100000 + int(r() * 380000) REQUESTS.append({"id": 4100 + i, "label": MAIN[i % 5][1], "a": a, "b": b, "main": MAIN[i % 5][0], "extra": EXTRA[(5 * i + 2) % 6], "text": f"{MAIN[i % 5][0]}, {EXTRA[(5 * i + 2) % 6]}." f" ana sayac {a} ara sayac {b}."}) def draw(task, seed, t): """Sampling table: one stream per task and seed, one value per request.""" if (task, seed) not in SAMPLES: r = rng(seed + 1009 * TASKS[task]) SAMPLES[(task, seed)] = [r() for _ in range(24)] return SAMPLES[(task, seed)][t["id"] - 4100] def call(task, t, seed, sampling=True): """Simulator: produces the response only from the token sequence. In tasks that choose among candidates, sampling can bring the second candidate; counting and exact arithmetic produce no candidates, they follow a single path over the token sequence.""" tokens = tokenize(t["text"]) if task == "counting": return tokens.count("saya") if task == "exact_arithmetic": p = [(str(t["a"])[k:k + 2], str(t["b"])[k:k + 2]) for k in range(0, 6, 2)] return int("".join(f"{(int(x) + int(y)) % 100:02d}" for x, y in p)) r = draw(task, seed, t) if task == "summarization": # which sentence enters the summary sentence = sorted([t["main"], t["extra"]], key=lambda c: -len(tokenize(c))) return sentence[1] if sampling and r < 0.14 else sentence[0] ranked = sorted(CLASSES, key=lambda s: (-sum(b in LEXICON[s][0] for b in tokens), s)) return ranked[1] if sampling and r < 0.14 else ranked[0] def measure(seed, sampling=True): d, ordered = {}, 0 d["summarization"] = sum(call("summarization", t, seed, sampling) == t["main"] for t in REQUESTS) / 24 d["classification"] = sum(call("classification", t, seed, sampling) == t["label"] for t in REQUESTS) / 24 for g in range(24): three = [REQUESTS[g], REQUESTS[(g + 7) % 24], REQUESTS[(g + 13) % 24]] truth = [t["id"] for t in sorted(three, key=lambda t: (LEXICON[t["label"]][1], t["id"]))] predicted = [t["id"] for t in sorted(three, key=lambda t: ( LEXICON[call("ranking", t, seed, sampling)][1], t["id"]))] ordered += truth == predicted d["ranking"] = ordered / 24 d["counting"] = sum(call("counting", t, seed) == t["text"].replace(",", " ") .replace(".", " ").split().count("sayac") for t in REQUESTS) / 24 d["exact_arithmetic"] = sum(call("exact_arithmetic", t, seed) == t["a"] + t["b"] for t in REQUESTS) / 24 return d RUN_A, RUN_B = measure(SEED), measure(SEED + 1) MEAN_TOKENS = sum(len(tokenize(t["text"])) for t in REQUESTS) / 24 print(f"fictional task set: {len(REQUESTS)} support requests, five task classes") print(f"input tokens per request {MEAN_TOKENS:.1f}, calls 1, latency class single call") print(f"{'task':<22}{'seed 20260218':<16}{'seed 20260219':<16}changed requests") for g in TASKS: diff = sum(str(call(g, t, SEED)) != str(call(g, t, SEED + 1)) for t in REQUESTS) print(f"{g:<22}{RUN_A[g]:<16.4f}{RUN_B[g]:<16.4f}{diff:>2}/24") ``` ``` fictional task set: 24 support requests, five task classes input tokens per request 33.3, calls 1, latency class single call task seed 20260218 seed 20260219 changed requests summarization 0.7500 0.6667 4/24 classification 0.8750 0.8750 2/24 ranking 0.6250 0.4583 11/24 counting 0.4167 0.4167 0/24 exact_arithmetic 0.2917 0.2917 0/24 ``` The number falls from class to class: classification **0.8750**, summarization **0.7500**, ranking **0.6250**, counting **0.4167**, exact arithmetic **0.2917**. The same model, the same request, the same token count. The only thing that changes is what the task asks for. Ranking giving the lowest number among the upper three rows has its own reason. Ranking is not a single decision but the combination of three: the simulator classifies each request separately and orders the triple by the priority of the classes. Because a mistake on a single request spoils the whole order, the hit rate of 0.8750 per class falls to 0.6250 per triple. The number of composite tasks is always lower than the number of their components, and this is a rule that will continue throughout this lesson. The upper three classes also share something: all of them are a **selection** task. Summarization chooses one of two sentences, classification one of five labels, ranking the order of three requests. In selection tasks the token surface carries enough information, because the correct answer is already somewhere inside the text. The lower two classes are **generation** tasks: the correct answer is not found inside the text, it has to be computed from the text. What shows the distinction, however, is not the measure itself but how the measure is taken. The last column says this alone. In the first three classes there are requests whose response changes when the seed changes: **11/24** in ranking, **4/24** in summarization, **2/24** in classification. The hit rate of classification comes out the same in the two runs but **not on the same requests**: two requests changed their response, one improved and one got worse. In the last two classes the difference is **0/24**. Counting and exact arithmetic give **exactly the same wrong answer** in both runs. This is not sampling noise. #### The Token-Level Source of Counting and Exact Arithmetic ```python def inflected(m): return [w for w in m.replace(",", " ").replace(".", " ").split() if w.startswith("sayac") and w != "sayac"] t = REQUESTS[4] print("request:", t["main"] + ",", t["extra"]) print("tokens:", " ".join(tokenize(t["main"]))) truth = t["text"].replace(",", " ").replace(".", " ").split().count("sayac") print(f"sayac occurrences: true {truth}, simulator {call('counting', t, SEED)}," f" inflected form {inflected(t['text'])}") print(f"addition: {t['a']} + {t['b']} = {t['a'] + t['b']}," f" simulator {call('exact_arithmetic', t, SEED)}") carries = sum(any(int(str(x["a"])[k:k + 2]) + int(str(x["b"])[k:k + 2]) > 99 for k in (2, 4)) for x in REQUESTS) inflected_count = sum(bool(inflected(x["text"])) for x in REQUESTS) print(f"requests with a carry across a piece boundary {carries}/24 -> remaining {24 - carries}/24") print(f"requests with an inflected sayac form {inflected_count}/24" f" -> remaining {24 - inflected_count}/24") ``` ``` request: abonelik devri sayaca islenmedi, kesinti suresi beklenenden uzun surdu tokens: abon elik devr i saya ca isle nmed i sayac occurrences: true 2, simulator 3, inflected form ['sayaca'] addition: 139039 + 302204 = 441243, simulator 431243 requests with a carry across a piece boundary 17/24 -> remaining 7/24 requests with an inflected sayac form 14/24 -> remaining 10/24 ``` Counting first. The model does not see words in the text; it sees pieces. The word `sayaca` (to the meter) is split into the pieces `saya` and `ca`, and the word `sayac` (meter) also starts with `saya`. Because the two share the same beginning in the token sequence, the model counts the inflected form as well and writes **3** instead of **2**. The agglutinative structure of Turkish makes this a rule, not an exception: requests carrying an inflected form of sayac are **14/24**, those not carrying one **10/24**. The counting hit rate is **0.4167**, that is exactly **10/24**. The model is wrong not because it cannot count but because **it cannot see the unit it is to count**. Exact arithmetic is the number side of the same flaw. The reading `139039` is split into the pieces `13`, `90`, `39`. The model adds the pieces pairwise and keeps each piece at two digits: when `90 + 22` comes to `112`, the third digit, that is the **carry**, cannot pass to the next piece because that piece has already been written. The number that comes out is **431243**, the correct one is **441243**: the difference is exactly the lost carry. Requests carrying a carry across a piece boundary are **17/24**, those not carrying one **7/24**. The exact-arithmetic hit rate is **0.2917**, that is exactly **7/24**. Both numbers come not from the data but from **tokenization**; changing the seed does not move them. #### Adding Scaffolding to the Collapsing Class ```python TRUTH = {"classification": lambda t: t["label"], "exact_arithmetic": lambda t: t["a"] + t["b"], "counting": lambda t: t["text"].replace(",", " ").replace(".", " ") .split().count("sayac")} def majority(task): # scaffolding 1: three runs, majority response s = 0 for t in REQUESTS: y = [call(task, t, k) for k in (SEED, SEED + 1, SEED + 2)] s += max(y, key=y.count) == TRUTH[task](t) return s / 24 def with_procedure(task, t): """Scaffolding 2: the model extracts the value, a procedure outside the tokens does the calculation.""" if task == "exact_arithmetic": numbers, o, tokens = [], 0, tokenize(t["text"]) while o < len(tokens): # adjacent digit pieces form one number p = "" while o < len(tokens) and tokens[o].isdigit(): p, o = p + tokens[o], o + 1 o += 1 if not p else 0 if p: numbers.append(int(p)) return sum(numbers[:2]) return sum(w == "sayac" for w in t["text"].replace(",", " ").replace(".", " ").split()) print(f"{'task':<18}{'bare call':<11}{'majority':<10}procedure") for g in ("classification", "counting", "exact_arithmetic"): y = ("-" if g == "classification" else f"{sum(with_procedure(g, t) == TRUTH[g](t) for t in REQUESTS) / 24:.4f}") print(f"{g:<18}{measure(SEED)[g]:<11.4f}{majority(g):<10.4f}{y}") extra_tokens = sum(len(tokenize(str(TRUTH['exact_arithmetic'](t)))) for t in REQUESTS) / 24 print(f"cost -- bare call: {MEAN_TOKENS:.1f} tokens, 1 call, latency class single call") print(f"cost -- majority: {3 * MEAN_TOKENS:.1f} tokens, 3 calls, latency class parallel") print(f"cost -- procedure: {2 * MEAN_TOKENS + extra_tokens:.1f} tokens, 2 calls, latency class chain") ``` ``` task bare call majority procedure classification 0.8750 0.9167 - counting 0.4167 0.4167 1.0000 exact_arithmetic 0.2917 0.2917 1.0000 cost -- bare call: 33.3 tokens, 1 call, latency class single call cost -- majority: 99.9 tokens, 3 calls, latency class parallel cost -- procedure: 69.6 tokens, 2 calls, latency class chain ``` The majority scaffolding calls the same request with three seeds and takes the most frequent response. It works in classification: the number **0.8750** rises to **0.9167**, because the error there comes from sampling and two of the three runs hold the correct answer. In counting and exact arithmetic, however, the number **does not budge**: 0.4167 and 0.2917 stay exactly the same. The cost is real and paid up front: the number of calls goes to three, the tokens to **99.9**. A statistical scaffolding placed on top of a structural error does nothing but repeat the error three times. The second scaffolding moves the work outside the token layer: the model only extracts the numbers, and a procedure does the adding and the counting. The hit rate is **1.0000** in both classes. The cost is two calls, **69.6** tokens and the **chain** latency class — because the second call has to wait for the result of the first, while the three calls of the majority scaffolding can go in parallel. The cheaper scaffolding is the one that pays off: majority gives nothing with 99.9 tokens, the procedure corrects the class entirely with 69.6 tokens. Without writing the cost these two scaffoldings could not have been told apart. #### Summary - When five task classes are measured with the bare call on the same fictional request set, the hit rate falls from 0.8750 to 0.2917: classification 0.8750, summarization 0.7500, ranking 0.6250, counting 0.4167, exact arithmetic 0.2917. - When the seed changes, there are requests whose response changes in the first three classes (11/24, 4/24, 2/24) and 0/24 in the last two; the error of counting and exact arithmetic is not sampling noise. - The counting hit rate equals exactly the 10/24 requests that carry no inflected form, the exact-arithmetic hit rate equals exactly the 7/24 requests that carry no carry across a piece boundary; both numbers come from tokenization. - The three-run majority scaffolding carries classification to 0.9167 but changes nothing in counting and exact arithmetic; its cost is 99.9 tokens and three calls. - The procedure scaffolding that takes the calculation out of the token layer carries both classes to 1.0000; its cost is 69.6 tokens, two calls and the chain latency class. #### Next Step The wrong answers measured in this lesson were honest in a way: when the model could not count it wrote a wrong number, and the wrongness was visible in a single line. The next lesson takes up a harder kind of wrong answer. To a question, even when it holds no basis whatsoever, the model produces a **fluent and confident** response; the form of the response is the same as the correct one, and its wrongness cannot be understood by looking at the response itself. The measure of this is not the hit rate but the **fabrication rate**: how many claims in the response rest on a document. The same lesson shows the second claim of the curriculum for the first time: one of the three scaffoldings added to reduce fabrication really lowers fabrication, **and lowers the hit rate too**, and this is not hidden. ### Hallucination Source: https://academia.sh/en/courses/introduction-to-ai-engineering/hallucination When twenty-four questions are asked over a fictional document pool of twenty-seven facts, the ungrounded response gets a hit rate of 0.2917 and a fabrication rate of 0.6667, and the response changes in 16/24 requests across two runs. Three mitigations are measured side by side: grounding raises the hit rate by +0.3750 and brings fabrication down to 0.3333, claim verification brings fabrication down to 0.0857 but its cost is two calls and 66.2 tokens, and the abstention option drops the hit rate 0.0417 below the baseline and leaves 10/24 requests unanswered. All of the unsupported claims left after grounding come from questions that have no counterpart in the document, and the verification scaffolding gives a difference of 0/24 across two runs. The wrong answers of the previous lesson were visible: when the model wrote a sum wrong, the result could be checked in a single line. The wrong answer of this lesson is not visible. Even when it has no basis whatsoever, the model produces a fluent and regular response; the form of a wrong response is the same as that of a correct one, it is written with the same certainty and cannot be told apart by looking at the response itself. This is called **hallucination**. Its measure is not the hit rate, because the hit rate asks only for the final value, and a response does not consist of a single value: the reasons, numbers and side information placed next to what was asked are also part of the response, and the user reads them too. The measure is therefore the **fabrication rate**: how many of the claims in the response rest on a document. This lesson puts three mitigation scaffoldings side by side on the same data, writes the cost of each as tokens and calls, and shows the curriculum's second claim for the first time — one of the three **lowers** the number, and this is not hidden in the table. - **LM52. The fictional document pool** consists of tariff descriptions, meter maintenance procedures and fault codes. Each fact is a `(topic, field, value)` triple and is **fictional**. - **LM53. Twenty-four questions are asked.** Some of the questions have a counterpart in the pool and some do not; this distinction is at the center of the measure. - **LM54. The simulator has two separate sources of knowledge:** the **document** placed inside the prompt and the **previously acquired fictional knowledge**. The second partly contradicts the pool. - **LM55. Each response carries two claims:** the field asked and a second field added next to it. A claim is **supported** if the exact same triple exists in the pool. - **LM56. Two numbers are written side by side.** The hit rate is the share of requests in which the value of the field asked agrees with the document; a request left unanswered does not count as a hit. The fabrication rate is the share of unsupported claims among all the claims produced. - **LM57. The uncertainty is measured with two runs.** A value read from the document is copied, a value coming from memory is **open to sampling**; the second run is taken with seed 20260219. #### Fabrication Rate in an Ungrounded Response ```python # COMMON -- FICTIONAL. The simulator does not call a real endpoint; the document pool, the questions # and the model's previously acquired knowledge are generated with the standard library, seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def tokenize(text): """Fictional tokenizer: letter pieces are split into fours, digit pieces into twos.""" out, buffer = [], "" for ch in text + " ": if ch.isalnum(): buffer += ch continue if buffer: n = 2 if buffer.isdigit() else 4 out += [buffer[i:i + n] for i in range(0, len(buffer), n)] buffer = "" if ch.strip(): out.append(ch) return out TOPICS = ["tariff home", "tariff business", "tariff garden", "meter maintenance", "meter replacement", "fault e12", "fault e31", "fault e44", "outage planned", "outage urgent", "subscription transfer", "subscription cancellation"] FIELDS = ["threshold", "duration", "step", "document"] DOCUMENT, MEMORY, CONFIDENCE, SAMPLES = {}, {}, {}, {} r = rng(SEED + 97) for k in TOPICS: # fictional document pool for a in FIELDS: p, v = r(), 2 + int(r() * 40) if p < 0.55: DOCUMENT[(k, a)] = v r = rng(SEED + 8080) for k in TOPICS: # the model's previously acquired fictional knowledge for a in FIELDS: p, v = r(), 2 + int(r() * 40) d = DOCUMENT.get((k, a)) MEMORY[(k, a)] = d if d is not None and p < 0.58 else v QUESTIONS = [(TOPICS[i % 12], FIELDS[(3 * i + 1) % 4]) for i in range(24)] r = rng(SEED + 4242) for i, (k, a) in enumerate(QUESTIONS): # fictional confidence: partly overlaps the truth CONFIDENCE[i] = (0.30 + 0.60 * r() if MEMORY[(k, a)] == DOCUMENT.get((k, a)) else 0.10 + 0.60 * r()) THRESHOLD = 0.55 def draw(i, j, seed): """Sampling table: a single draw for every value not read from the document.""" if seed not in SAMPLES: r2 = rng(seed + 5551) SAMPLES[seed] = [r2() for _ in range(48)] return SAMPLES[seed][2 * i + j] def ask(i, scaffolding, seed): """The response is a list of claims; each claim is a (topic, field, value) triple.""" k, a = QUESTIONS[i] side = FIELDS[(FIELDS.index(a) + 2) % 4] if scaffolding == "abstention" and CONFIDENCE[i] < THRESHOLD: return [] source = DOCUMENT if scaffolding in ("grounding", "verification") else {} claims = [] for j, x in enumerate((a, side)): if (k, x) in source: # read from the document in the prompt claims.append((k, x, source[(k, x)])) continue v = MEMORY[(k, x)] # a value coming from memory is open to sampling claims.append((k, x, 2 + (v + 7) % 40 if draw(i, j, seed) < 0.25 else v)) if scaffolding == "verification": # second call: each claim is looked up in the document r2 = rng(SEED + 611 * i) claims = [c for c in claims if DOCUMENT.get((c[0], c[1])) == c[2] or r2() < 0.10] return claims def measure(scaffolding, seed=SEED): claim_count = unsupported = hit = unanswered = 0 for i, (k, a) in enumerate(QUESTIONS): y = ask(i, scaffolding, seed) claim_count += len(y) unanswered += not y unsupported += sum(DOCUMENT.get((c[0], c[1])) != c[2] for c in y) hit += DOCUMENT.get((k, a)) is not None and any( c[1] == a and c[2] == DOCUMENT[(k, a)] for c in y) return hit / 24, unsupported / claim_count if claim_count else 0.0, claim_count, unanswered COVERED = sum((k, a) in DOCUMENT for k, a in QUESTIONS) QUESTION_TOKENS = sum(len(tokenize(f"what is the {a} value for {k}")) for k, a in QUESTIONS) / 24 DOC_TOKENS = sum(len(tokenize(f"{k} {a} value {v}")) for (k, a), v in DOCUMENT.items()) / len(TOPICS) hit1, fab1, n1, _ = measure("bare call") hit2, fab2, n2, _ = measure("bare call", SEED + 1) print(f"fictional document pool {len(DOCUMENT)} facts, {len(QUESTIONS)} questions," f" with a counterpart in the document {COVERED}/24") print(f"bare call, seed 20260218: hit rate {hit1:.4f}, fabrication {fab1:.4f}, {n1} claims") print(f"bare call, seed 20260219: hit rate {hit2:.4f}, fabrication {fab2:.4f}, {n2} claims") changed = sum(ask(i, "bare call", SEED) != ask(i, "bare call", SEED + 1) for i in range(24)) print(f"number of requests whose response changed in the two runs {changed}/24") print(f"cost: {QUESTION_TOKENS:.1f} tokens per request, 1 call, latency class single call") ``` ``` fictional document pool 27 facts, 24 questions, with a counterpart in the document 16/24 bare call, seed 20260218: hit rate 0.2917, fabrication 0.6667, 48 claims bare call, seed 20260219: hit rate 0.3333, fabrication 0.6667, 48 claims number of requests whose response changed in the two runs 16/24 cost: 12.1 tokens per request, 1 call, latency class single call ``` The bare call answers all twenty-four requests and produces **48 claims**. **0.6667** of them rest on no document. The hit rate is **0.2917**. What to notice is that the two numbers can move independently of each other: in the second run the hit rate rises to **0.3333** while the fabrication rate stays at **0.6667**. The hit rate measures a single field, the fabrication rate measures the whole response. There is not a single request that the model refuses to answer. Even though **8** of the twenty-four questions have no counterpart at all in the pool, all eight have a response. This shows the source of hallucination directly: the simulator does not look for what was asked in the document; it produces the most probable continuation from the **previously acquired fictional knowledge**, and that knowledge is both incomplete and partly contradictory to the pool. The fluency of the text is independent of the existence of a basis. The term itself calls for a warning here. Hallucination is **not the name of a malfunction**: the model fabricates not because it is broken but because it is doing exactly the job it was designed for. The job it does is to produce the probable continuation of a text, and probability is not the same thing as a basis. In the table the counterpart of this is the following: no scaffolding that lowers the fabrication rate touches the inside of the model; all of them put a source of basis or a checking step **around** the response. If this distinction is not held, fabrication is taken to be "a flaw that will be fixed one day" and the cost of the scaffolding is never written into the budget. #### Three Mitigation Scaffoldings ```python print(f"{'scaffolding':<14}{'hit rate':<10}{'fabrication':<13}{'claims':<8}{'unanswered':<12}" f"{'tokens':<8}{'calls':<7}latency") for d, tok, calls, lat in (("bare call", QUESTION_TOKENS, 1, "single call"), ("grounding", QUESTION_TOKENS + DOC_TOKENS, 1, "single call"), ("abstention", QUESTION_TOKENS + 9, 1, "single call"), ("verification", 2 * (QUESTION_TOKENS + DOC_TOKENS), 2, "chain")): hit, fab, n, un = measure(d) un_text = f"{un}/24" print(f"{d:<14}{hit:<10.4f}{fab:<13.4f}{n:<8}{un_text:<12}{tok:<8.1f}{calls:<7}{lat}") t1, t2 = measure("grounding")[0], measure("abstention")[0] print(f"baseline bare call {hit1:.4f}; grounding {t1 - hit1:+.4f}, abstention {t2 - hit1:+.4f}," f" verification {measure('verification')[0] - hit1:+.4f}") ``` ``` scaffolding hit rate fabrication claims unanswered tokens calls latency bare call 0.2917 0.6667 48 0/24 12.1 1 single call grounding 0.6667 0.3333 48 0/24 33.1 1 single call abstention 0.2500 0.5714 28 10/24 21.1 1 single call verification 0.6667 0.0857 35 2/24 66.2 2 chain baseline bare call 0.2917; grounding +0.3750, abstention -0.0417, verification +0.3750 ``` **Grounding** puts the relevant document inside the prompt. The hit rate rises by **+0.3750**, and the fabrication rate is cut in half. The cost is honest and is measured in tokens: **33.1** tokens per request instead of 12.1, that is, close to three times. The number of calls does not change. **Claim verification** checks, in a second call, each claim of the response against the pool one by one and drops the claim whose counterpart cannot be found. The fabrication rate goes down to **0.0857**, the lowest number in this table. Its cost is also the highest: **66.2** tokens, **two calls** and the **chain** latency class, because the second call has to wait for the output of the first. Its hit rate stays the same as grounding's, so the extra cost buys only fabrication. The **abstention option** is the third, and in the table it is **the row that lowers the number**. The hit rate goes down to **0.2500**: **0.0417 below** the baseline, the bare call. The reason is written in the unanswered column — the scaffolding leaves ten requests unanswered, and some of them were requests that would have been answered correctly. The fabrication rate goes down only slightly, to **0.5714**, because the fictional confidence measure overlaps with the existence of a basis only partly. The scaffolding takes away something definite from the hit rate, and brings something uncertain in return for fabrication. Had the number not been written, all three mitigations would have been listed as "improvements". This does not mean the scaffolding is bad; it means **the criterion was not chosen**. If the cost of stating a wrong tariff threshold is heavier than the cost of stating nothing, the abstention option wins and the loss of 0.0417 in the hit rate is cheap insurance. In the opposite case the same scaffolding is a regression that leaves ten requests unanswered. The engineer looking at the table has the job of converting these two costs into the same unit; the table's job is to make visible which number the decision rests on. The threshold itself is also a setting: a value lower than 0.55 reduces the unanswered requests and enlarges fabrication. The unanswered column stays in the table for this reason — if that column were not written, where the drop in the hit rate comes from would not be visible, and the scaffolding would be shut down as "it did not work". #### Where Fabrication Comes From ```python print(f"{'scaffolding':<14}{'hit rate A':<12}{'hit rate B':<12}{'changed requests':<18}" f"source of unsupported claims") for d in ("bare call", "grounding", "abstention", "verification"): a, b = measure(d)[0], measure(d, SEED + 1)[0] diff = sum(ask(i, d, SEED) != ask(i, d, SEED + 1) for i in range(24)) diff_text = f"{diff}/24" covered = uncovered = 0 for i in range(24): for c in ask(i, d, SEED): if DOCUMENT.get((c[0], c[1])) != c[2]: covered += (c[0], c[1]) in DOCUMENT uncovered += (c[0], c[1]) not in DOCUMENT print(f"{d:<14}{a:<12.4f}{b:<12.4f}{diff_text:<18}with document {covered:>2}," f" without document {uncovered:>2}") ``` ``` scaffolding hit rate A hit rate B changed requests source of unsupported claims bare call 0.2917 0.3333 16/24 with document 16, without document 16 grounding 0.6667 0.6667 8/24 with document 0, without document 16 abstention 0.2500 0.3333 9/24 with document 7, without document 9 verification 0.6667 0.6667 0/24 with document 0, without document 3 ``` The last column splits fabrication in two. **16** of the bare call's 32 unsupported claims are in questions that **have** a counterpart in the pool: the document was in our hands, and the model did not look at it. The other **16** are in questions that **do not have** a counterpart. Grounding **zeroes** the first half and does not touch the second half at all. This is the exact description of what grounding can and cannot do: it ends the fabrication of information that stands in the pool, it does not end the fabrication of information that does not stand in the pool. For the remaining sixteen claims the only solution is to enlarge the pool or to leave them unanswered. The middle column measures uncertainty. The bare call gives a different response in **16/24** requests across two runs. Grounding brings this down to **8/24**, because a value read from the document is copied and is not open to sampling. Claim verification gives **0/24**: exactly the same response in two runs. What narrows the uncertainty is not the setting of the model but **the response being tied to a basis**. This is the second and little-discussed gain of grounding. A design rule comes out of this, and it reaches ahead in this curriculum. The cheap way of reducing fabrication is not to force the model but **to put the basis of the response into the prompt**; the expensive way is to check the response after producing it. When the cost of the two is written in the same unit, in tokens and calls, the choice is not left to debate. But both of them leave a question unanswered: if there are hundreds of documents in the pool, which ones will be put into the prompt. In this lesson the correct document was given by hand every time, because the number of topics was twelve. When the number of documents grows, finding the correct document becomes a scaffolding that itself needs to be measured, and it brings its own hit rate and its own token cost. A later course of this curriculum builds exactly that scaffolding; this lesson only showed with numbers why it is needed. #### Summary - In the ungrounded response, 24 requests produce 48 claims, the fabrication rate is 0.6667 and the hit rate 0.2917; the model answers all eight of the questions that have no counterpart in the pool. - Grounding raises the hit rate by +0.3750 and brings fabrication down to 0.3333; its cost is going from 12.1 tokens per request to 33.1 tokens. - Claim verification pulls fabrication lowest, to 0.0857; its cost is 66.2 tokens, two calls and the chain latency class; it adds nothing to the hit rate on top of grounding. - The abstention option drops the hit rate to 0.2500, that is 0.0417 below the baseline, and leaves 10/24 requests unanswered; not every scaffolding that is added improves. - Grounding brings fabrication from 16 to 0 in claims that have a document but leaves 16 as 16 in those without one; it also pulls the requests that change from run to run from 16/24 to 8/24. #### Next Step Every scaffolding measured in this lesson was built **around** the model: a document was placed inside the prompt, a second call was added, a threshold was defined. All of them assumed that access to the model was given. Yet that access is itself a decision and it changes the numbers. Between reaching a model only through a remote endpoint and holding the weights and running them on your own hardware, which setting stays with whom, how many tokens the same job costs and where the data goes are different. The next lesson compares these two access modes on three criteria with numbers and writes in which criterion the choice changes. ### Closed- and Open-Weight Models Source: https://academia.sh/en/courses/introduction-to-ai-engineering/closed-and-open-weight-models Two access modes are compared on three criteria with numbers, and the criteria do not converge on a single form. The control held is 3/9 settings in the closed-weight form and 9/9 in the open-weight form. On unit cost the closed form holds a hit rate of 0.7917 at 41.8 tokens per request, while the open form reaches the same hit rate only after six examples are added, at 179.8 tokens, and the ratio comes out at 4.30. On the data boundary the closed form sends 24/24 requests, 1004 tokens and 72 numeric fields across the organization boundary, and the open form sends none. Two criteria pick the open-weight form, one picks the closed-weight form, and the three numbers have no common unit. Every scaffolding added in the previous lesson was built around the model, and access to the model was assumed to be given. That access is itself a decision. A language model can be reached only through a remote **model endpoint**; or the model's **weights** are held and run on your own hardware. The first is called **closed-weight** access and the second **open-weight** access. This distinction is usually discussed as a debate over preference, and the debate most often ends where both sides are right, because the sides speak from different criteria. This lesson turns it into numbers on three criteria: the number of settings that stay in hand, the token cost of finishing the same job, and the data that crosses the organization boundary. All three are quantities an engineer can count from their own side, without ever looking inside the model. No model, provider or license name appears, because what is measured is not a product but **the structure of the access mode**; names change in five years, the structure does not. - **LM58. The task set has the same structure and the simulator is the same as before**: 24 fictional support requests, now with English texts, are divided into five classes, and the measure is the hit rate. The requests, the classes and the readings are **fictional**. - **LM59. The two access modes are modeled with a single difference:** the **width of the simulator's familiar piece set**. It is wide in the closed form and narrow in the open form. This is the assumption of the fiction and not a claim about the capability of real models; what is measured is **how the difference turns into cost**. - **LM60. The scaffolding of the open form is adding examples.** Each fictional example placed in the prompt costs 23 tokens and widens the familiar piece set by one step. - **LM61. The control held is measured in a fictional list of nine settings**, by counting in which form each setting can be decided. - **LM62. The data boundary is measured with three numbers:** the number of requests that cross the organization boundary, the number of tokens and the number of numeric fields. The numeric fields are fictional subscriber numbers and meter readings; this criterion does not count the nature of the data, it counts the amount that crosses the boundary. - **LM63. Uncertainty is measured with two runs**; the second run is taken with seed 20260219. Because the two forms share the same simulator, they are expected to give the same number at the same width, and this is used as a checkpoint in the table. - **LM64. The criteria are read separately and not added up.** The number of settings, the number of tokens and the number of requests crossing the boundary are not converted into a common unit; the winner of each criterion is written separately. #### Unit Cost: What the Same Hit Rate Costs ```python # SHARED -- FICTIONAL. Both access modes are built on the same simulator; no model, provider or # license is named. The only difference in the fiction is the width of the familiar piece set. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def tokenize(text): """Fictional tokenizer: letter pieces are split into fours, digit pieces into twos.""" out, buffer = [], "" for ch in text + " ": if ch.isalnum(): buffer += ch continue if buffer: n = 2 if buffer.isdigit() else 4 out += [buffer[i:i + n] for i in range(0, len(buffer), n)] buffer = "" if ch.strip(): out.append(ch) return out MAIN = [("meter not read estimated invoice issued", "reading"), ("invoice amount doubled compared to last month", "payment"), ("water pressure dropped during the night", "water"), ("planned outage notice did not arrive", "outage"), ("subscription transfer was not applied to meter", "subscription")] EXTRA = ["meter number may have been changed", "address information looks old", "same problem did not occur last month", "cannot reach the meter door closed", "outage lasted longer than expected", "no automatic payment instruction on meter"] LEXICON = {"outage": ["outa", "noti"], "reading": ["read", "meas", "esti"], "water": ["pres", "drop", "tank", "pipe"], "payment": ["invo", "bill", "paym", "amou"], "subscription": ["subs", "addr", "tran", "cont"]} CLASSES = list(LEXICON) EXAMPLE_TOKENS = 23 # cost of each fictional example added to the prompt REQUESTS = [] for i in range(24): r = rng(SEED + 31 * i) a, b = 100000 + int(r() * 380000), 100000 + int(r() * 380000) REQUESTS.append({"id": 4100 + i, "label": MAIN[i % 5][1], "text": f"subscriber {4100 + i}: {MAIN[i % 5][0]}, {EXTRA[(5 * i + 2) % 6]}." f" main meter {a} sub meter {b}."}) def call(t, width, seed): """Simulator: picks a label from the pieces it knows; returns ambiguous if the top two tie.""" tokens, known = tokenize(t["text"]), {s: set(o[:width]) for s, o in LEXICON.items()} p = sorted(((sum(b in known[s] for b in tokens), s) for s in CLASSES), key=lambda x: (-x[0], x[1])) if p[0][0] == 0 or p[0][0] == p[1][0]: return "ambiguous" # both labels are written at once, the format check fails r = rng(seed + 1009 * width + 7 * t["id"]) return p[1][1] if r() < 0.14 else p[0][1] def measure(width, examples, seed=SEED): correct = token_count = 0 for t in REQUESTS: correct += call(t, width, seed) == t["label"] token_count += len(tokenize(t["text"])) + examples * EXAMPLE_TOKENS return correct / 24, token_count / 24 print("access mode examples hit rate A hit rate B calls tokens/request") for name, w, e in (("closed-weight", 4, 0), ("open-weight", 1, 0), ("open-weight", 2, 2), ("open-weight", 3, 4), ("open-weight", 4, 6)): a, tokens = measure(w, e) b = measure(w, e, SEED + 1)[0] print(f"{name:<16}{e:<10}{a:<12.4f}{b:<12.4f}{'1.0000':<8}{tokens:.1f}") ``` ``` access mode examples hit rate A hit rate B calls tokens/request closed-weight 0 0.7917 0.7083 1.0000 41.8 open-weight 0 0.1250 0.1667 1.0000 41.8 open-weight 2 0.3750 0.3750 1.0000 87.8 open-weight 4 0.5833 0.6667 1.0000 133.8 open-weight 6 0.7917 0.7083 1.0000 179.8 ``` In the bare call the two forms spend the same number of tokens and do not give the same number: closed **0.7917**, open **0.1250**. The difference comes not from the model itself but from the narrow piece set of the simulator that runs in the open form; this is the assumption of the fiction. What matters is what follows: the difference is **one that can be closed**, and the price of closing it is written in tokens. As two examples are added to the prompt at a time, the hit rate climbs the steps **0.3750**, **0.5833** and **0.7917**. At the sixth example the open form catches up exactly with the number of the closed form: **0.7917** against **0.7917**, and in the second run **0.7083** against **0.7083**. The two forms now do the same job. The only thing they do not do is do it at the same price: **41.8** tokens per request against **179.8** tokens. The number of calls is one in both, and the latency class is a single call in both; the difference lies entirely in the length of the prompt. The place to read in the ladder is this: **the cost moved from the model to the prompt**. In the open-weight form, what the model does not know is retold in every request. This is the appearance in this lesson of the general law of scaffolding: what is missing in one place is paid for in tokens in another place, and that payment is repeated in every request. In a set of twenty-four requests the difference is 3,312 tokens; as the number of requests grows the difference grows in direct proportion, because unless the examples are cached they are sent again in every call. There is also a number that does not appear in the table. When the scores of the top two candidates are equal, the simulator writes both labels at once instead of a single label and fails the format check; these responses are counted as misses. With the narrow piece set this happens often, with the wide set it is rare. So part of the low number of the open form comes not from a wrong label but from **unusable output**. When the measure is reduced to a single number this distinction disappears, yet the two kinds of failure mean different things on the product side: a wrong label silently starts a wrong job, while a malformed output is a request that stops. #### Control Held and the Data Boundary ```python SETTINGS = [("the weights themselves", 0, 1), ("tokenizer", 0, 1), ("version pinning", 0, 1), ("context window limit", 0, 1), ("rate limit", 0, 1), ("rollback during an outage", 0, 1), ("distribution temperature", 1, 1), ("stop condition", 1, 1), ("output length limit", 1, 1)] CLOSED, OPEN = sum(c for _, c, _ in SETTINGS), sum(o for _, _, o in SETTINGS) print(f"{'setting':<26}{'closed-weight':<15}open-weight") for name, c, o in SETTINGS: print(f"{name:<26}{'held' if c else '-':<15}{'held' if o else '-'}") print(f"criterion 1 -- settings held: closed {CLOSED}/{len(SETTINGS)}, open {OPEN}/{len(SETTINGS)}") ids = sum(sum(w.isdigit() for w in t["text"].replace(":", " ").replace(".", " ").split()) for t in REQUESTS) outside = sum(len(tokenize(t["text"])) for t in REQUESTS) print(f"criterion 3 -- beyond the boundary: closed {len(REQUESTS)}/{len(REQUESTS)} requests," f" {outside} tokens, {ids} numeric fields") print(f"{'':36}open 0/{len(REQUESTS)} requests, 0 tokens, 0 numeric fields") ``` ``` setting closed-weight open-weight the weights themselves - held tokenizer - held version pinning - held context window limit - held rate limit - held rollback during an outage - held distribution temperature held held stop condition held held output length limit held held criterion 1 -- settings held: closed 3/9, open 9/9 criterion 3 -- beyond the boundary: closed 24/24 requests, 1004 tokens, 72 numeric fields open 0/24 requests, 0 tokens, 0 numeric fields ``` **Three** of the nine settings stay in hand under closed-weight access: the distribution temperature, the stop condition and the output length limit. These are the settings sent along with the request. The remaining **six** are not in hand, and they all have something in common: the weights, the tokenizer, the version, the window limit, the rate limit and returning to the old behavior at the moment of an outage are decisions that **belong to the place where the model runs**. If the model runs on someone else's machine, those decisions are with someone else. Under open-weight access all nine stay in hand, because the party that runs the model and the party that decides are the same. An invisible consequence of this is that the version cannot be pinned. The fabrication rate measured in the seventh lesson, the task hit rate measured in the sixth lesson and the 0.7917 measured in this lesson all belong to a particular behavior. When that behavior changes without notice, all the measures must be taken again and none of the old numbers is valid. Under closed-weight access the lifetime of a measurement is bounded by the lifetime of a decision that is not in hand. The third criterion is read in the same table. In the closed form, **24** of the twenty-four requests, **1004** tokens and **72** numeric fields cross the organization boundary; these numeric fields are fictional subscriber numbers and meter readings. In the open form all three numbers are zero. This criterion is not a difference of degree between the two but **a difference of threshold**: the data that crosses the boundary either exists or it does not. The table should not be read one way only. In return for the six settings that are not in hand under closed-weight access, there is a load that does not pass to the organization: running the model, setting aside hardware, moving the version and absorbing the outage. Under open-weight access these nine settings stay in hand because **all nine responsibilities stay in hand**. When the criterion is named "control held" this symmetry becomes invisible; next to the number, what it brings must be written as much as what it burdens. What this lesson measures is not which form is good, but **the criteria on which the two forms part**. #### Where the Choice Changes ```python cb, ob = measure(4, 0), measure(4, 6) print("criterion closed-weight open-weight winner") print(f"control {CLOSED}/9 settings {OPEN}/9 settings open") print(f"unit cost {cb[1]:.1f} tokens {ob[1]:.1f} tokens closed") print(f"data boundary 24/24 requests 0/24 requests open") print(f"for the same hit rate {cb[0]:.4f} the cost ratio is {ob[1] / cb[1]:.2f}") print(f"two criteria pick the open-weight form, one picks the closed-weight form;" f" no common unit") ``` ``` criterion closed-weight open-weight winner control 3/9 settings 9/9 settings open unit cost 41.8 tokens 179.8 tokens closed data boundary 24/24 requests 0/24 requests open for the same hit rate 0.7917 the cost ratio is 4.30 two criteria pick the open-weight form, one picks the closed-weight form; no common unit ``` The choice **changes on the unit cost criterion**. While control and the data boundary pick the open-weight form, unit cost picks the closed form, and the difference is not small: **4.30 times** the tokens for the same hit rate. Because two criteria are on one side and one on the other, saying "the majority picks open" would be wrong, since the three numbers **have no common unit**. The number of settings and the number of tokens cannot be added; the number of requests crossing the boundary cannot be compared with either of the two. This is not a deficiency but the correct reading of the measurement. The decision is made by **weighting** the criteria, and what gives the weight is the context of the system. In an organization that cares less about the latency of the response than about where the data goes, the third criterion alone decides and the 4.30-fold cost is accepted. In an organization with a tight token budget the second criterion dominates. The same three numbers support two different decisions as the weights change; what changes is not the measurement but the weight. The rule of the course works here for the second time: a given component is measured from its surface. None of the numbers measured in this lesson came from inside the model. The number of settings was read from what the interface allows, the number of tokens from what was put in the prompt, the data that crosses the boundary from where the request goes. All three are traces left outside by a component whose inside cannot be seen, and all three are things an engineer can count from their own code. This is the gain of writing these three numbers instead of calling an access mode "stronger" or "safer": descriptions are argued over, numbers are compared. And when the cost is not written next to the number, the comparison turns back into an argument — 0.7917 looks the same in both rows, and the 4.30-fold difference between them exists only in the cost column. #### Summary - Closed-weight access leaves 3 of the nine settings in hand and open-weight access leaves 9; all six settings that are not in hand belong to the place where the model runs. - In the bare call the hit rate is 0.7917 against 0.1250; when six examples are added to the prompt the open form catches up with the number of the closed form in both runs (0.7917 and 0.7083). - The cost of the same hit rate is 41.8 tokens per request against 179.8 tokens, a ratio of 4.30 times; the number of calls and the latency class are the same in both. - On the data boundary, in the closed form 24/24 requests, 1004 tokens and 72 numeric fields cross the organization boundary and in the open form none does; this is a difference of threshold, not of degree. - The choice changes on the unit cost criterion, and because the three numbers have no common unit, the decision is made by weighting. #### Next Step This lesson compared two forms and stopped where the decision is left to the weights. In a real choice there are not two options and there are not three criteria: within the same access mode there are candidates that differ from one another on the axes of capability, latency and token cost, and privacy stands above them as a separate criterion. The next lesson builds three candidates at the simulator level, counts them on four criteria and answers a single question: when the weights change, does the winner change, and if it does, at which threshold? The weight sweep and the normalization procedure were measured in the Environment Selection lesson of the Data Analysis with Python course; they will not be repeated here, only their result will be used. ### Model Selection Source: https://academia.sh/en/courses/introduction-to-ai-engineering/model-selection Three candidates are built at the simulator level and counted on four criteria: small capability 0.5000 hit rate / 89.5 equivalent tokens / 7.29 steps / 0 requests crossing, medium capability 0.5833 / 195.3 / 7.00 / 0, large capability 0.7917 / 670.0 / 14.00 / 24. Across five weight sets the winner is medium three times, large once and small once. The weight sweep gives the thresholds: when the hit-rate weight reaches 4 the choice moves from medium to large, when the cost weight reaches 2 it moves from medium to small, and the latency and data-boundary weights do not change the choice up to twelve. When the same computation is repeated with seed 20260219, the winner of the hit-rate weighted set turns from large back to medium. The previous lesson compared two access modes on three criteria and stopped where the decision is left to the weights. In a real choice there are not two options. There are several models that could do the same job; they differ from one another on the axes of capability, token cost, latency and where the data goes, and none of them is best on all four axes at once. This lesson builds three candidates **at the simulator level**, counts them on four criteria and answers a single question: when the weights change, does the winner change, and if it does, at which threshold? How a weight sweep and the choice of a normalization rule can turn the result was measured in the Environment Selection lesson of the Data Analysis with Python course; that procedure is not repeated here, a single normalization rule is used and its result is read. - **LM65. The three candidates are three settings of a single simulator.** They differ in the width of the familiar piece set, the unit multiplier and the length of the response they produce. No model, version or provider is named; the candidates are called **small**, **medium** and **large capability** and are **fictional**. - **LM66. The cost unit is the equivalent token.** A token of a candidate is scaled by a fictional multiplier that says how many times the account of the smallest candidate's token it holds: small 1, medium 4, large 12. No currency is written. - **LM67. Latency is measured in the number of steps:** the average number of calls per request times the number of output tokens produced. Raw time is not written, because generation is sequential and the number of steps does not change from run to run. - **LM68. The small and medium candidates run on their own hardware with open-weight access**, and the large candidate is called only through a model endpoint; the number of requests crossing the organization boundary is 0, 0 and 24 respectively. - **LM69. The small candidate has a scaffolding:** for responses that fail the format check, two fictional examples are added to the prompt and the request is asked again. This scaffolding raises the hit rate and enlarges the number of calls and tokens. - **LM70. The normalization rule is single:** in each criterion the worst candidate gets 0 and the best candidate gets 1, and the values in between are scaled to the range. The weights are applied to these normalized numbers. #### Three Candidates and Four Criteria ```python # SHARED -- FICTIONAL. The three candidates are built on the same simulator; no model or # provider is named. The candidates differ in familiar piece width, unit multiplier and output # length. The cost unit is EQUIVALENT TOKENS: number of tokens times the candidate's unit multiplier. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def tokenize(text): """Fictional tokenizer: letter pieces are split into fours, digit pieces into twos.""" out, buffer = [], "" for ch in text + " ": if ch.isalnum(): buffer += ch continue if buffer: n = 2 if buffer.isdigit() else 4 out += [buffer[i:i + n] for i in range(0, len(buffer), n)] buffer = "" if ch.strip(): out.append(ch) return out MAIN = [("meter not read estimated invoice issued", "reading"), ("invoice amount doubled compared to last month", "payment"), ("water pressure dropped during the night", "water"), ("planned outage notice did not arrive", "outage"), ("subscription transfer was not applied to meter", "subscription")] EXTRA = ["meter number may have been changed", "address information looks old", "same problem did not occur last month", "cannot reach the meter door closed", "outage lasted longer than expected", "no automatic payment instruction on meter"] LEXICON = {"outage": ["outa", "noti"], "reading": ["read", "meas", "esti"], "water": ["pres", "drop", "tank", "pipe"], "payment": ["invo", "bill", "paym", "amou"], "subscription": ["subs", "addr", "tran", "cont"]} CLASSES = list(LEXICON) REQUESTS = [] for i in range(24): r = rng(SEED + 31 * i) a, b = 100000 + int(r() * 380000), 100000 + int(r() * 380000) REQUESTS.append({"id": 4100 + i, "label": MAIN[i % 5][1], "text": f"subscriber {4100 + i}: {MAIN[i % 5][0]}, {EXTRA[(5 * i + 2) % 6]}." f" main meter {a} sub meter {b}."}) # (name, piece width, unit multiplier, output tokens, example cost, requests crossing the boundary) CANDIDATES = [("small", 2, 1, 5, 46, 0), ("medium", 3, 4, 7, 0, 0), ("large", 4, 12, 14, 0, 24)] def call(t, width, seed): """Simulator: picks a label from the pieces it knows; returns ambiguous if the top two tie.""" tokens, known = tokenize(t["text"]), {s: set(o[:width]) for s, o in LEXICON.items()} p = sorted(((sum(b in known[s] for b in tokens), s) for s in CLASSES), key=lambda x: (-x[0], x[1])) if p[0][0] == 0 or p[0][0] == p[1][0]: return "ambiguous" r = rng(seed + 1009 * width + 7 * t["id"]) return p[1][1] if r() < 0.14 else p[0][1] def measure(candidate, seed=SEED): name, width, multiplier, output, example, outside = candidate correct = calls = inputs = 0 for t in REQUESTS: n = len(tokenize(t["text"])) y, calls, inputs = call(t, width, seed), calls + 1, inputs + n if y == "ambiguous" and example: # scaffolding: add two examples to the prompt and ask again y, calls, inputs = call(t, width + 1, seed), calls + 1, inputs + n + example correct += y == t["label"] calls, inputs = calls / 24, inputs / 24 return {"name": name, "hit_rate": correct / 24, "equivalent": (inputs + calls * output) * multiplier, "steps": calls * output, "calls": calls, "outside": outside} S = [measure(c) for c in CANDIDATES] print("candidate hit rate hit rate B equivalent tokens steps calls/request crossing") for c, s in zip(CANDIDATES, S): b = measure(c, SEED + 1)["hit_rate"] print(f"{s['name']:<11}{s['hit_rate']:<10.4f}{b:<12.4f}{s['equivalent']:<19.1f}" f"{s['steps']:<7.2f}{s['calls']:<15.4f}{s['outside']}/24") ``` ``` candidate hit rate hit rate B equivalent tokens steps calls/request crossing small 0.5000 0.5833 89.5 7.29 1.4583 0/24 medium 0.5833 0.6667 195.3 7.00 1.0000 0/24 large 0.7917 0.7083 670.0 14.00 1.0000 24/24 ``` The four columns show four different winners. On hit rate the **large** candidate leads: 0.7917. On equivalent tokens the **small** candidate leads: 89.5, that is, less than a seventh of the large one's 670.0. On steps the **medium** candidate leads, but the margin is thin: 7.00 against 7.29. On requests crossing the organization boundary the small and medium candidates lead together, because both run on their own hardware. Where the small candidate loses on the number of steps is written in the calls column: **1.4583** calls per request. This is the cost of the scaffolding. For responses that fail the format check, two examples are added to the prompt and the request is asked again; the hit rate rises to 0.5000 this way, but every request that is asked again adds one more step. The scaffolding takes the hit rate and gives up latency, and both numbers stand side by side in the table. Attention must be paid to what the equivalent token unit is. The three candidates receive the same request and read almost the same number of tokens; by raw token count all three are close to one another. The difference lies not in the **number** of the token but in what it **holds**: processing the same token means twelve times the computational load on the large candidate compared with the small one. The unit multiplier reduces this load to a single number and makes the comparison possible. This is also the reason no currency is written: the multiplier is structural and reads the same way five years from now, whereas a price list does not. The output tokens are counted separately, because the output is produced sequentially and enters both the cost and the steps; a 14-token response from the large candidate means twice the latency of the small candidate, which gives the same response in 5 tokens. #### When the Weight Changes, the Winner Changes ```python CRITERIA = [("hit rate", "hit_rate", 1), ("equivalent tokens", "equivalent", -1), ("steps", "steps", -1), ("requests crossing", "outside", -1)] def normalize(results, key, direction): """Single normalization rule: in the range the worst is 0, the best is 1.""" v = [s[key] for s in results] lo, hi = min(v), max(v) if hi == lo: return [1.0] * len(v) return [((x - lo) if direction > 0 else (hi - x)) / (hi - lo) for x in v] N = {k: normalize(S, k, d) for _, k, d in CRITERIA} WEIGHTS = [("balanced", (1, 1, 1, 1)), ("hit-rate weighted", (4, 1, 1, 1)), ("cost weighted", (1, 4, 1, 1)), ("latency weighted", (1, 1, 4, 1)), ("data-boundary weighted", (1, 1, 1, 4))] print(f"{'criterion':<20}{'small':<8}{'medium':<8}large") for name, k, _ in CRITERIA: print(f"{name:<20}{N[k][0]:<8.4f}{N[k][1]:<8.4f}{N[k][2]:.4f}") print(f"{'weight set':<24}{'small':<8}{'medium':<8}{'large':<8}winner") for name, w in WEIGHTS: p = [sum(w[j] * N[k][i] for j, (_, k, _) in enumerate(CRITERIA)) for i in range(3)] print(f"{name:<24}{p[0]:<8.4f}{p[1]:<8.4f}{p[2]:<8.4f}{S[p.index(max(p))]['name']}") ``` ``` criterion small medium large hit rate 0.0000 0.2857 1.0000 equivalent tokens 1.0000 0.8176 0.0000 steps 0.9583 1.0000 0.0000 requests crossing 1.0000 1.0000 0.0000 weight set small medium large winner balanced 2.9583 3.1033 1.0000 medium hit-rate weighted 2.9583 3.9605 4.0000 large cost weighted 5.9583 5.5562 1.0000 small latency weighted 5.8333 6.1033 1.0000 medium data-boundary weighted 5.9583 6.1033 1.0000 medium ``` Across five weight sets, three different winners come out. With equal weights the **medium** candidate wins, and the margin is small: 3.1033 against 2.9583. When the hit-rate weight is raised to four, the **large** candidate moves ahead, but again by a hair: 4.0000 against 3.9605. When the cost weight is raised to four, the **small** candidate wins. The latency and data-boundary weights do nothing other than enlarge the medium candidate. The same three candidates, the same four numbers, three different decisions. The only thing that changes is which criterion counts for how much. For this reason the question "which model is better" has no measure; the question that has a measure is "with these four numbers and these weights, which candidate wins". A justification for a choice, as long as it does not write its weights, is not a justification but a preference. The normalization rule is itself a decision, and it does not show in the table. The rule used here scales to the range: in each criterion the worst candidate gets 0 and the best gets 1. This rule does not look at how large the difference in between is — on hit rate the 0.0833 difference between small and medium is divided by the 0.2917 width of the range and rises to 0.2857, that is, a small absolute difference turns into a large normalized difference. Another rule would have given other numbers. Because the Environment Selection lesson measured that this can turn the choice, a single rule is used here and the fact that it is used is written; when it is not written, the table looks more objective than it is. #### The Threshold and the Choice's Own Uncertainty ```python def winner(N, w): p = [sum(w[j] * N[k][i] for j, (_, k, _) in enumerate(CRITERIA)) for i in range(3)] return CANDIDATES[p.index(max(p))][0] print(f"{'swept criterion':<20}{'weight 1':<10}{'first change':<14}new winner") for j, (name, _, _) in enumerate(CRITERIA): before, threshold, new = winner(N, (1, 1, 1, 1)), "-", "-" for w in range(2, 13): k = winner(N, tuple(w if i == j else 1 for i in range(4))) if k != before and threshold == "-": threshold, new = w, k print(f"{name:<20}{before:<10}{str(threshold):<14}{new}") S2 = [measure(c, SEED + 1) for c in CANDIDATES] N2 = {k: normalize(S2, k, d) for _, k, d in CRITERIA} print(f"{'weight set':<24}{'seed 20260218':<16}seed 20260219") for name, w in WEIGHTS: print(f"{name:<24}{winner(N, w):<16}{winner(N2, w)}") ``` ``` swept criterion weight 1 first change new winner hit rate medium 4 large equivalent tokens medium 2 small steps medium - - requests crossing medium - - weight set seed 20260218 seed 20260219 balanced medium medium hit-rate weighted large medium cost weighted small small latency weighted medium medium data-boundary weighted medium medium ``` The thresholds are asymmetric. When the cost weight is **2**, the choice moves from medium to small; the hit-rate weight does not bring the large candidate ahead until it reaches **4**. Even when the latency and data-boundary weights are swept up to twelve, they do not change the choice, because on these two criteria the winner is already medium and raising the weight does nothing other than reinforce the winner. Enlarging the weight of a criterion never brings ahead a candidate that is not ahead on that criterion. The last table is the most uncomfortable number of this lesson. When the same computation is repeated with seed 20260219, **the winner of the hit-rate weighted set turns from large back to medium**. No criterion changed, no weight changed; only the sampling seed of the simulator changed, and the hit rate of the large candidate fell from 0.7917 to 0.7083. A decision won by a hair, when it is built on a component that changes from run to run, becomes dependent on the run itself. In three of the four criteria the choice is solid; in the fourth it is not, and this is something a comparison measured with a single run could never see. A practical rule follows from this. A selection table should write not only the winner but also the **winning margin**. Rows with a wide margin stand even if the weights wobble a little or the measurement drifts a little; those with a thin margin do not. The margin of 0.1450 between 3.1033 and 2.9583 at equal weights and the margin of 0.0395 between 4.0000 and 3.9605 in the hit-rate weighted set stand in the same table but do not say the same thing. In a row with a thin margin the right behavior is not to declare the winner but to run both candidates again by enlarging the measurement, or to tie the decision to one more criterion. #### Summary - When three candidates are counted on four criteria the winner splits four ways: large on hit rate with 0.7917, small on equivalent tokens with 89.5, medium on steps with 7.00, small and medium on requests crossing the organization boundary with 0/24. - The small candidate's scaffolding raises the hit rate to 0.5000 while taking the number of calls to 1.4583 and the steps to 7.29; the gain and the cost stand in the same table. - Across five weight sets the winner is medium three times, large once and small once; at equal weights the margin is thin, like 3.1033 against 2.9583. - The thresholds are asymmetric: a cost weight of 2 turns the choice from medium to small, and a hit-rate weight of 4 turns it from medium to large; the latency and data-boundary weights do not change the choice up to twelve. - With seed 20260219 the winner of the hit-rate weighted set turns from large back to medium; a choice given by a hair becomes dependent on the run. #### Next Step In this lesson cost was counted as a criterion, but it was never used as a limit. The winning candidate was chosen freely under every weight set; yet in a real system the token budget is an upper bound and imposes not the choice but the **design**. Even with the same task, the same model and the same hit rate, how many tokens are spent per request varies greatly with how the prompt is written. The next lesson compares four scaffoldings as tokens and hit rate — a long prompt, a short prompt with examples, summarize and send, and a cache — and shows which design becomes mandatory when the budget narrows. That lesson is also the close of the course. ### Token-Based Cost Source: https://academia.sh/en/courses/introduction-to-ai-engineering/token-based-cost Over a fictional stream of ninety-six requests, four scaffoldings are compared in tokens and hit rate: the long prompt gives 0.7083 hit rate at 159.8 tokens per request, the short prompt with examples gives the same hit rate at 103.8 tokens, summarize and send gets down to 74.0 tokens with 0.8021, the hard summary drops the hit rate to 0.0833 at 68.0 tokens, and the cache gives 0.7500 at 17.7 tokens. The total tokens fall from 15,341 to 1,702 and the model calls fall from 96 to 23. When the budget drops to 7,000 tokens only two of the five scaffoldings fit, and the design is not chosen but imposed. The hit rate of the cache comes out at 0.7500 and 0.9583 across the two runs. In the previous lesson the token was counted as a criterion but was never used as a limit. In a real system the budget is an upper bound and imposes not the choice but the **design**. Even with the same task, the same model and the same hit rate, the tokens spent per request change several times over depending on how the prompt is written. This lesson compares four scaffoldings on the same request stream, as tokens and hit rate. The unit is the **token**; no currency is written, because the number of tokens is measurable and lasting, while a price is neither measurable nor lasting. - **LM71. The request stream is 96 fictional support requests** and contains repetition: 24 requests are drawn so that their frequencies are unequal. The requests and the classes are **fictional**, the seed is **20260218**. - **LM72. The measure is twofold:** tokens per request (small is good) and hit rate (large is good). The number of model calls is written separately. - **LM73. The four scaffoldings are like a ladder**, each row adds something to the one before: long prompt, short prompt with examples, summarize and send, cache. The fourth row, the hard summary, is a harder setting of the third row and is measured to show the limit. - **LM74. Summarize and send trims the request before putting it into the prompt.** The trimming is done before the model, on the organization's own side, and makes the model charge no tokens at all. - **LM75. The cache stores the response by request.** When the same request comes a second time the model is not called at all; if the stored response is correct it is repeated as correct, if it is wrong it is repeated as wrong. - **LM76. Uncertainty is measured with two runs**; the second run is taken with seed 20260219. #### Tokens and Hit Rate of the Four Scaffoldings ```python # SHARED -- FICTIONAL. The simulator does not call a real endpoint. The cost unit is the TOKEN; # no currency is written. The request stream is 96 fictional support requests with repetition. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def tokenize(text): """Fictional tokenizer: letter pieces are split into fours, digit pieces into twos.""" out, buffer = [], "" for ch in text + " ": if ch.isalnum(): buffer += ch continue if buffer: n = 2 if buffer.isdigit() else 4 out += [buffer[i:i + n] for i in range(0, len(buffer), n)] buffer = "" if ch.strip(): out.append(ch) return out MAIN = [("meter not read estimated invoice issued", "reading"), ("invoice amount doubled compared to last month", "payment"), ("water pressure dropped during the night", "water"), ("planned outage notice did not arrive", "outage"), ("subscription transfer was not applied to meter", "subscription")] EXTRA = ["meter number may have been changed", "address information looks old", "same problem did not occur last month", "cannot reach the meter door closed", "outage lasted longer than expected", "no automatic payment instruction on meter"] LEXICON = {"outage": ["outa", "noti"], "reading": ["read", "meas", "esti"], "water": ["pres", "drop", "tank", "pipe"], "payment": ["invo", "bill", "paym", "amou"], "subscription": ["subs", "addr", "tran", "cont"]} CLASSES = list(LEXICON) REQUESTS = [] for i in range(24): r = rng(SEED + 31 * i) a, b = 100000 + int(r() * 380000), 100000 + int(r() * 380000) REQUESTS.append({"id": 4100 + i, "label": MAIN[i % 5][1], "text": f"subscriber {4100 + i}: {MAIN[i % 5][0]}, {EXTRA[(5 * i + 2) % 6]}." f" main meter {a} sub meter {b}."}) r = rng(SEED + 9091) STREAM = [int(r() ** 1.7 * 24) for _ in range(96)] # fictional request stream with repetition # (name, instruction tokens, example tokens, input trim, cache) SCAFFOLDINGS = [("long prompt", 118, 0, 0, False), ("short prompt + example", 16, 46, 0, False), ("summarize and send", 16, 46, 12, False), ("hard summary", 16, 46, 6, False), ("cache", 16, 46, 12, True)] def call(t, trim, order, seed): """Simulator: uses the tokens visible in the prompt; trimming shortens the visible part.""" tokens = tokenize(t["text"]) tokens = tokens[:trim] if trim else tokens known = {s: set(o[:4]) for s, o in LEXICON.items()} p = sorted(((sum(b in known[s] for b in tokens), s) for s in CLASSES), key=lambda x: (-x[0], x[1])) if p[0][0] == 0 or p[0][0] == p[1][0]: return "ambiguous" return p[1][1] if rng(seed + 37 * order + 7 * t["id"])() < 0.14 else p[0][1] def measure(scaffolding, seed=SEED): name, instruction, example, trim, cached = scaffolding cache, correct, tokens, calls = {}, 0, 0, 0 for order, i in enumerate(STREAM): t = REQUESTS[i] if cached and i in cache: correct += cache[i] == t["label"] continue # the request never goes to the model n = len(tokenize(t["text"])) y = call(t, trim, order, seed) tokens += instruction + example + (min(n, trim) if trim else n) calls += 1 cache[i] = y correct += y == t["label"] return correct / len(STREAM), tokens / len(STREAM), calls, tokens print(f"fictional request stream {len(STREAM)} requests, unique requests {len(set(STREAM))}/24") print("scaffolding hit rate hit rate B tokens/request model calls total tokens") for s in SCAFFOLDINGS: h, b, c, total = measure(s) h2 = measure(s, SEED + 1)[0] print(f"{s[0]:<24}{h:<10.4f}{h2:<12.4f}{b:<16.1f}{f'{c}/96':<13}{total}") ``` ``` fictional request stream 96 requests, unique requests 23/24 scaffolding hit rate hit rate B tokens/request model calls total tokens long prompt 0.7083 0.7396 159.8 96/96 15341 short prompt + example 0.7083 0.7396 103.8 96/96 9965 summarize and send 0.8021 0.8438 74.0 96/96 7104 hard summary 0.0833 0.0833 68.0 96/96 6528 cache 0.7500 0.9583 17.7 23/96 1702 ``` The first two rows give the same hit rate: **0.7083**, and in the second run **0.7396**. The only difference between them is that two examples are put in place of a long instruction, with a short instruction. The difference in cost is **56.0** tokens per request, **5,376** tokens in total. This is the easiest gain in the table, and it makes the hit rate pay nothing. The long prompt looks like a good prompt for as long as it is not measured. The third row goes in an unexpected direction: summarize and send brings the tokens down to **74.0** while **raising** the hit rate to **0.8021**. The reason is in the fiction and can be read — the trimming drops the extra clause of the request, and the extra clause carried the pieces that blurred the class evidence. This is not a general law, it is a property of this data set. Indeed the fourth row shows the limit: when the trimming is hardened to six tokens the hit rate collapses to **0.0833**, and in return only **6.0** tokens are saved. A summary is a setting, and the good side of the setting and its cliff are very close to each other. The last row carries the largest gain in the table: **17.7** tokens per request, model calls **23/96**. Seventy-three of the ninety-six requests never go to the model. The hit rate is **0.7500**, below the third row, but the two-run columns must be looked at: the same cache gives **0.9583** with the second seed. The cache neither lowers nor raises the hit rate; it **freezes** it. Whatever the first response to a request turned out to be, all the repetitions of that request receive that response. A cache placed in front of an uncertain component does not remove the uncertainty; it reduces it to a single draw and makes the result permanent. #### The Design the Budget Imposes ```python RESULTS = [(s[0],) + measure(s) for s in SCAFFOLDINGS] print("budget per request scaffoldings that fit design imposed by the budget hit rate") for budget in (16000, 10000, 7000, 2000, 1500): fitting = [s for s in RESULTS if s[4] <= budget] best = max(fitting, key=lambda s: s[1]) if fitting else None print(f"{budget:<8}{budget / 96:<13.1f}{f'{len(fitting)}/5':<23}" f"{(best[0] if best else 'none'):<30}{(f'{best[1]:.4f}' if best else '-')}") print("scaffolding without a written cost: long prompt and short prompt + example give the same hit rate," " difference 5376 tokens") ``` ``` budget per request scaffoldings that fit design imposed by the budget hit rate 16000 166.7 5/5 summarize and send 0.8021 10000 104.2 4/5 summarize and send 0.8021 7000 72.9 2/5 cache 0.7500 2000 20.8 1/5 cache 0.7500 1500 15.6 0/5 none - scaffolding without a written cost: long prompt and short prompt + example give the same hit rate, difference 5376 tokens ``` When the budget is wide the design is **chosen**: at 16,000 tokens all five of the five scaffoldings fit and the one that gives the highest hit rate is selected. When the budget drops to 7,000 the design is **imposed**: only two scaffoldings fit, and the best is not 0.8021 but **0.7500**. The 0.0521 in between is not an engineering error, it is the price of the budget. At 1,500 tokens no scaffolding fits; at that point what to do is not to write a better prompt but either to enlarge the budget or to keep part of the requests from passing through the model at all. The order in which this table is read also matters. As the budget narrows, the first scaffolding to fall is the most expensive one, not the worst one. The long prompt, at 15,341 tokens, is the first to leave the list, and its hit rate is exactly the same as the short prompt's; so the budget first cuts **the cost that buys nothing**. This is also the first step in making a system cheaper. A scaffolding whose cost is not written cannot be defended, because it has not been measured. #### Summary - The long prompt and the short prompt plus examples give the same hit rate (0.7083); the difference is 56.0 per request and 5,376 tokens in total, and it makes the hit rate pay nothing. - Summarize and send brings the tokens down to 74.0 while raising the hit rate to 0.8021, but when the trimming is hardened to six tokens the hit rate collapses to 0.0833 and the return is only 6.0 tokens. - The cache is the cheapest at 17.7 tokens per request and never sends 73 of the 96 requests to the model; it neither lowers nor raises the hit rate, it reduces it to a single draw and freezes it (0.7500 and 0.9583). - When the budget drops to 7,000 tokens two of the five scaffoldings fit and the design is imposed rather than chosen; the best reachable hit rate falls from 0.8021 to 0.7500. - The first scaffolding to leave as the budget narrows is the most expensive one and it contributes nothing to the hit rate; the first step in making a system cheaper is removing the scaffoldings whose gain has not been measured. #### Course Wrap-Up The accounting of fourteen lessons is in a single table, and every number was read from its own lesson. The rows do not share a common unit; the second column states **by which measure** each row is read. **The difference is given in a single sign convention: the number of the scaffolding minus the number of the bare call**, in parentheses in the third column. | Lesson | Bare call | With scaffolding | Cost | |---|---|---|---| | Field 01 Who Is an AI Engineer | label hit rate 0.5167 | label set and format constraint 0.7750 (+0.2583) | total tokens 1.49 times, calls 120 → 124, per correct label 69.6 → 69.2 | | Field 02 The Difference Between AI Engineering and Machine Learning Engineering | hit rate without format constraint 0.5000 | format constraint 0.8056 (+0.3056) | −634 tokens, 0 labeled examples; retry +1 call, +51 tokens, (+0.0000) | | Field 03 Impact on Product Development | hit rate without scaffolding 0.5139 | grounding 0.8264 (+0.3125) | +5,914 tokens, 144 retrieval steps, latency class chain | | Field 04 The Distinction Between AI and General AI | tested among general claims 0/9 | tested among measurable claims 5/5, passed 3 (+5 tests) | four components per claim: task, metric, set, threshold with its direction | | Language 01 Tokens | 27.78 tokens per request, hit rate 0.9722 | 0.50 threshold 11.68 tokens, 0.9722 (−16.10 tokens) | preprocessing step and frequency table; 0.30 threshold drops the hit rate to 0.8611 | | Language 02 Transformer Intuition | last 16 tokens, hit rate 0.0000 | distance-independent weight 1.0000 (+1.0000) | comparisons 16 → 70.9; at 1024 tokens 8.26 times the fixed window | | Language 03 Pretraining and Fine-Tuning | fault code hit rate 0.0900 | fine-tuning with 400 examples 0.9000 (+0.8100) | 400 labeled examples; general text 0.3186 → 0.2692 (−0.0494) | | Language 04 Inference | output limit 4, hit rate 0.0000 | limit 8, hit rate 1.0000 (+1.0000) | 4.00 → 6.43 tokens, 126.0 → 211.1 comparisons (1.68 times) | | Language 05 Context Window | full response 0.3833, prompt 282.3 tokens | dropping from the head 0.7433 (+0.3600) | −64.8 tokens but an extra truncation step; timeout share 0.0200 → 0.0600 | | Language 06 Capabilities and Limits | counting 0.4167, exact arithmetic 0.2917 | procedure scaffolding 1.0000 and 1.0000 (+0.5833, +0.7083) | 33.3 → 69.6 tokens, 1 → 2 calls, chain; majority at 99.9 tokens (+0.0000) | | Language 07 Hallucination | fabrication rate 0.6667, hit rate 0.2917 | grounding 0.3333 and 0.6667 (−0.3334, +0.3750) | 12.1 → 33.1 tokens; abstention option lowers the hit rate (−0.0417) | | Language 08 Closed- and Open-Weight Models | open-weight bare hit rate 0.1250 | with six examples 0.7917 (+0.6667) | 41.8 → 179.8 tokens (4.30 times); in the closed form 24/24 requests cross the organization boundary | | Language 09 Model Selection | medium candidate at balanced weights 3.1033 | large candidate at hit-rate weight 4 4.0000 (+0.8967) | equivalent tokens 195.3 → 670.0 (3.43 times), steps 7.00 → 14.00 | | Language 10 Token-Based Cost | long prompt 0.7083, 159.8 tokens per request | cache 0.7500, 17.7 tokens (+0.0417, −142.1) | model calls 96 → 23; hit rate 0.7500 and 0.9583 across two runs | **This is the rule of the course: a given component is measured from its surface; the number of a component whose inside cannot be seen is written on its surface.** None of the numbers in the table came from inside the model. All of them were taken by counting where a request went in, where a response came out, and how many tokens, how many calls and how many steps were spent in between. The model's architecture, its weights and the data it was trained on appear nowhere in the table, because these were not measurable quantities in this course; everything that was measurable was on the surface. **The rule of the curriculum can also be read from here: the number of an artificial intelligence system is not the number of the model but the number of the scaffolding; a scaffolding whose cost is not written counts as unmeasured.** This is why the fourth column stands in the table and is empty in no row. The second claim of the curriculum can also be read by name from the same table: **adding scaffolding does not always improve.** In Field 02 the retry moved the number by 0.0000 and added one call and 51 tokens. In Field 03 offering options raised the visible change from 0.3056 to 0.4306. In Language 01 the 0.30 threshold lowered the hit rate from 0.9722 to 0.8611. In Language 03 fine-tuning fixed the domain text while bringing the general text down from 0.3186 to 0.2692. In Language 05 summarizing brought the prompt down to 100.6 tokens but, because it dropped the step that was asked about, collapsed the full-response rate from 0.3833 to 0.0000. In Language 06 the three-run majority spent 99.9 tokens and changed nothing in counting and exact arithmetic. In Language 07 the abstention option dropped the hit rate 0.0417 below the baseline. In Language 10 the hard summary collapsed the hit rate from 0.8021 to 0.0833. Eight rows, a single rule: the cost of a scaffolding is certain, its gain is not. In fourteen lessons the surface of the model was measured and the cost was written next to every number. The only thing not asked is **what the model was told**. In the whole table the prompt was either held fixed or mentioned by its length; what is written in it, in which order, where an example is placed and how the instruction is built was never a variable. How much the number moves when the same model is given a different text on the same task, and that the text itself is a **design object**, was not measured. The next course, Prompt Engineering, opens exactly this variable.