--- title: 'Prompt Engineering' source: 'https://academia.sh/en/courses/prompt-engineering' language: en updated: '2026-10-21T05:02:33+00:00' license: 'CC BY-SA 4.0' lessons: 22 outcomes: - 'Writes prompts systematically by separating them into input, context and instruction components and measures the token share of each component' - 'Applies reasoning and example-based techniques to suitable tasks and writes the cost of a technique next to its gain' - 'Shows with a number how much sampling parameters and output control narrow the uncertainty' --- # Prompt Engineering Measuring, when the model, the task and the set are fixed and the only thing that changes is the text, how much each text decision moves the number and how much it costs: the token share of the three components of a prompt, the cost of the system instruction repeated in every request, comparing the difference that specifying a role adds with the difference between two runs, the response distribution that an ambiguous request produces, the parseability of a format constraint, the point where the example count saturates and the effect of order, the case where stating a prohibition increases violations, overfitting to the validation set while iterating, chain of thought multiplying tokens on a single-step task without a gain, the call cost in self-consistency and agreement breaking, the cost of branching, the cost of invisible intermediate steps, errors overlapping in a prompt ensemble, sampling settings narrowing the difference between two runs, the rate of responses that match the schema and what retrying recovers, whether the same error is fixed by an instruction or by missing information, and the subgroup gap that balancing through the prompt does not close. ## Prompt Design Splitting a prompt into its components and measuring the cost of each design decision next to its number: the token share of input, context and instruction and what happens when a component is removed, the total that repeating the system instruction in every request inflates and the accuracy that shortening it gives up, comparing the difference that role prompting adds with the difference between two runs, in how many different forms an ambiguous prompt is answered over ten runs, the share of parseable responses under a format constraint and the cost of a length limit, the point where the gain saturates as the example count goes from zero to eight and the effect of example order, the case where a negative constraint increases violations and its comparison with a positive statement, how the validation and test numbers diverge as the iteration count grows. ### What Is a Prompt Source: https://academia.sh/en/courses/prompt-engineering/what-is-a-prompt Splitting a prompt into input, context and instruction components and measuring the token share of each: across 120 fictional support tickets the instruction holds 0.3945, the context 0.3114 and the input 0.2941. When a single component is removed, the number and the cost are read side by side. The full prompt gives a label hit rate of 0.7833 and 5974 tokens; with the context removed it gives 0.7083 and 4177, with the instruction removed 0.7167 and 4751, with the input removed 0.1417 and 4197. The two-run difference is 0.0583 for the full prompt and rises to between 0.2000 and 0.2583 for prompts missing the context or the instruction. The previous course measured the surface of the model across fourteen lessons and wrote the cost next to every number: how many tokens went in, how many tokens came out, where the limit was, what the same input gave in two runs. At its close, one thing was missing. Throughout all of those measurements, **what was said** to the model stayed the same. The instruction text was written once and then treated like a fixed background; how much the number would move if the same model were given a different text for the same task was never asked. This course turns that text into an object of measurement. The model, the task and the ticket set are fixed throughout the course; the only thing that changes is **the text**. In every lesson three numbers sit side by side: the previous prompt, the modified prompt with a single design decision applied, and the cost of the change. The cost is written in **tokens**; token cost and budget were measured in the closing lesson of the previous course and are not repeated here. **The rule of the course is this: a prompt is a design object; a prompt changed without measuring is not an improvement but a guess.** This lesson takes that object apart. - **PD1.** The code does not call a real model endpoint; it calls a **simulator** written with the standard library. The simulator is the same definition built in the previous course: a fictional tokenizer, a 240-token context window, a 24-token maximum output, seed-dependent nondeterminism, a rate limit and a timeout. - **PD2.** The support tickets are **fictional**: 120 records, five topics, a misleading sentence appended to some of the records. The topic names come from the fictional document vocabulary of the field team. - **PD3.** The prompt is split into three components, and the split follows the **function** of the text. The **instruction** says what to do, the **context** defines the world in which the decision is made, the **input** carries the record to which that decision will be applied. - **PD4.** The criterion is the **label hit rate**. If no label can be read from a response, the record counts as wrong and is also shown separately in the **unlabeled response share**. - **PD5.** The cost is tokens, and **input and output are written separately**. The two do not move in the same direction; this is one of the main findings of this lesson. - **PD6.** The contribution of a component is measured by removing that component. The other two components, the model, the seed and the ticket set do not change. - **PD7.** Every measurement is run with two seeds (20260218 and 20260219). The seed is not a setting; it is the place where nondeterminism is read. - **PD8.** Token, context window and inference were defined in the language models topic of the previous course; here they are only used. #### The Three Components of a Prompt From the outside, a prompt looks like a single text. From the inside there are three separate texts doing three separate jobs, and the cost and the contribution of each are different. The **instruction** defines the task: assign the ticket to a label, give the response in this form. The **context** sets up the world in which the decision is made: what the label set is, which options exist. The **input** is the only part that changes in each request: the ticket to be classified itself. The instruction and the context stay the same across one hundred twenty requests; the input changes one hundred twenty times. The first block sets up the simulator and the fiction, then writes these three components separately and counts their tokens. ```python # SIMULATOR. This code does not call a real model endpoint; it is a language model # simulator written with the standard library. The support tickets are FICTIONAL too. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUT = 240, 24 # context window and maximum output, in tokens def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer: pieces of at most five characters b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def prompt_trace(m): # trace that depends on the prompt content, independent of the run h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] TEMPLATES = { "fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"], } EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim", "acil bilgi almam gerekiyor", "kayit numaram elimde yok"] TICKETS = [] for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = TEMPLATES[k][int(r() * 3)] if r() < 0.34: m += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": m, "topic": k}) CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "borc": "odeme", "odedi": "odeme", "ucret": "odeme", "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet", "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"} TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} # misleading cues FREE_FORM = ["talebiniz {k} konusuyla ilgili gorunuyor", "bu kayit {k} basligina girer , ilgili ekibe yonlendirildi", "kaydiniz alindi , konuyu inceleyip donus yapacagiz"] CALL_COUNT = [0] def call(prompt, seed=SEED, temperature=0.6, max_out=MAX_OUT): CALL_COUNT[0] += 1 tok = tokenize(prompt) if len(tok) > WINDOW: return {"status": "context_overflow", "tokens_in": len(tok), "tokens_out": 0, "response": ""} if CALL_COUNT[0] % 37 == 0: return {"status": "rate_limit", "tokens_in": len(tok), "tokens_out": 0, "response": ""} r = rng(seed ^ prompt_trace(prompt)) if len(tok) > 90 and r() < 0.04: return {"status": "timeout", "tokens_in": len(tok), "tokens_out": 0, "response": ""} cand = [k for k in CONCEPTS if k[:5] in tok] # concepts named in the prompt if len(cand) < 3: cand = CONCEPTS p = {k: 0.15 for k in cand} for t in tok: for table, w in ((CUES, 1.0), (TRAPS, 1.3)): if table.get(t) in p: p[table[t]] += w if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) # most likely choice else: e = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [v / sum(e) for v in e])] y = k if "tek" in tok else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k) ot = tokenize(y) if len(ot) > max_out: # stop condition ot = ot[:max_out] y = " ".join(ot) return {"status": "ok", "tokens_in": len(tok), "tokens_out": len(ot), "response": y} def read_label(y): # extract a label from the response for s in y.lower().split(): if s in TOPICS: return s return None INSTRUCTION = "Talebi asagidaki etiketlerden birine ata . Yanit olarak tek etiket yaz ." CONTEXT = "Etiketler: fatura , ariza , sayac , tarife , kesinti ." def build_prompt(x, instruction=True, context=True, inp=True): # builds a prompt from chosen components p = [] if instruction: p.append(INSTRUCTION) if context: p.append(CONTEXT) if inp: p.append("Talep: " + x["text"]) return " ".join(p) x0 = TICKETS[0] print(f"FICTIONAL ticket set: {len(TICKETS)} records, {len(TOPICS)} topics") for name, m in (("instruction", INSTRUCTION), ("context", CONTEXT), ("input", "Talep: " + x0["text"])): print(f"{name:<11} {len(tokenize(m)):>2} tokens | {m}") p0 = build_prompt(x0) print(f"{'full prompt':<11} {len(tokenize(p0)):>2} tokens") for sd in (SEED, SEED + 1): print(f"seed {sd}: {call(p0, seed=sd)['response']}") ``` ``` FICTIONAL ticket set: 120 records, 5 topics instruction 19 tokens | Talebi asagidaki etiketlerden birine ata . Yanit olarak tek etiket yaz . context 15 tokens | Etiketler: fatura , ariza , sayac , tarife , kesinti . input 13 tokens | Talep: planli kesinti duyurusu nereden takip edilir full prompt 47 tokens seed 20260218: kesinti seed 20260219: kesinti ``` Only thirteen of the forty-seven tokens of the prompt belong to the real job, the ticket to be classified. The remaining thirty-four are fixed text that is sent again in every request. The two seeds gave the same response, `kesinti`, to the same prompt; this does not mean that nondeterminism has disappeared, it means that two runs coincided on a single record. Throughout this course this comparison is made on **the whole set**. #### The Token Share of Each Component The cost of a component is not its own length but **its length multiplied by the number of requests**. Its contribution can be measured only by removing it. The second block does both: first it computes the share of the three components in total input tokens, then it drops one component at a time and runs the same 120 records again. ```python def measure(seed=SEED, **kw): CALL_COUNT[0] = 0 ti = to = d = unl = 0 for x in TICKETS: s = call(build_prompt(x, **kw), seed=seed) ti, to = ti + s["tokens_in"], to + s["tokens_out"] e = read_label(s["response"]) unl += e is None d += e == x["topic"] n = len(TICKETS) return {"hit_rate": d / n, "unlabeled": unl / n, "tokens_in": ti, "tokens_out": to} n_ins, n_ctx = len(tokenize(INSTRUCTION)), len(tokenize(CONTEXT)) n_inp = sum(len(tokenize("Talep: " + x["text"])) for x in TICKETS) / len(TICKETS) total = n_ins + n_ctx + n_inp print(f"{'component':<12} {'tokens':>9} {'share':>8}") for name, v in (("instruction", n_ins), ("context", n_ctx), ("input", n_inp), ("total", total)): print(f"{name:<12} {v:>9.2f} {v / total:>8.4f}") print() CASES = (("full prompt", {}), ("no context", {"context": False}), ("no instruction", {"instruction": False}), ("no input", {"inp": False})) print(f"{'prompt':<14} {'hit rate':>8} {'run 2':>8} {'diff':>7} {'unlabeled':>10} " f"{'in':>6} {'out':>6} {'total':>7}") for name, kw in CASES: A, B = measure(**kw), measure(seed=SEED + 1, **kw) print(f"{name:<14} {A['hit_rate']:>8.4f} {B['hit_rate']:>8.4f} " f"{abs(A['hit_rate'] - B['hit_rate']):>7.4f} {A['unlabeled']:>10.4f} {A['tokens_in']:>6d} " f"{A['tokens_out']:>6d} {A['tokens_in'] + A['tokens_out']:>7d}") ``` ``` component tokens share instruction 19.00 0.3945 context 15.00 0.3114 input 14.17 0.2941 total 48.17 1.0000 prompt hit rate run 2 diff unlabeled in out total full prompt 0.7833 0.7250 0.0583 0.0250 5780 194 5974 no context 0.7083 0.5083 0.2000 0.0667 3980 197 4177 no instruction 0.7167 0.4583 0.2583 0.2167 3500 1251 4751 no input 0.1417 0.2167 0.0750 0.0250 4080 117 4197 ``` The first table gives the proportion to remember throughout the course. The real data, the ticket to be classified, holds **0.2941 of the prompt**. The instruction and the context together hold **0.7059**, and these two texts are identical letter for letter across one hundred twenty requests. The average ticket is 14.17 tokens, the fixed text is 34 tokens: in every request, 2.4 times as much fixed text as data is sent again. #### When a Component Is Removed The second table gives the contribution of the components and shows that the three do not do the same job. **When the context is removed**, the hit rate drops from 0.7833 to 0.7083 and total tokens drop from 5974 to 4177, that is, by 1797 tokens. At first glance the gain and the loss look balanced. But the unlabeled response share has risen from 0.0250 to 0.0667: when the label set is not written in the prompt, the model also answers with words outside the five topics, and no label can be read from those responses. The job of the context is not to make the model smarter but to **close the answer space**. **When the instruction is removed**, the hit rate drops to 0.7167 and input tokens fall from 5780 to 3500 — and output tokens **jump from 194 to 1251**. This is the harshest row of the table. The instruction was the longest component at nineteen tokens; dropping it gained 2280 tokens on input, but took back 1057 tokens on output, and the net gain fell to 1223 tokens. Because the responses turned into sentences, the unlabeled share rose from 0.0250 to **0.2167**: one in every five responses became unreadable by machine. In this prompt the instruction is not the most expensive component per unit length but **the one with the best return**. **When the input is removed**, the number that comes out is the most instructive row of this lesson. The prompt is still valid, it still spends 4197 tokens, the unlabeled response share is still 0.0250 — that is, the output looks trouble-free. The hit rate, however, collapses to 0.1417; because the same prompt goes to all one hundred twenty records, the model produces the same guess every time, and that guess is right only as often as the share of that label in the set. **A prompt produces a well-formed and confident response even without carrying data.** This is why judging a prompt by its form does not count as measurement. #### A Prompt Changed Without Measuring Is a Guess The `diff` column of the table runs the third rule of the course for the first time. The full prompt gave 0.7833 and 0.7250 on the two seeds; **the two-run difference is 0.0583**. The effect of removing the context on the hit rate is 0.0750, and the effect of removing the instruction is 0.0667. Both numbers sit just above the two-run difference and are not large enough to trust on their own. In this course such a number **counts as unmeasured**; the rule is applied by name in the following lessons. The measured side of the same rows is elsewhere. The unlabeled response share rose from 0.0250 to 0.2167, output tokens grew sixfold, and the **own two-run difference** of the prompts missing the context or the instruction rose to between 0.2000 and 0.2583. The three components stabilize the hit rate as much as they add to it: while the full prompt moves 0.0583 from run to run, these two prompts move three to four times as much. The components of a prompt do more to **narrow the range of freedom** of the model than to give it information; the gain comes from there. The cost is also written apart from this reading. The full prompt spends 5974 / 0.7833 / 120, that is, 63.6 tokens per correct label. The prompt without context spends 49.1 tokens, the prompt without instruction 55.2 tokens. So the most accurate prompt is **also the most expensive prompt**, and these two must stand on separate rows. Removing a component lowers the cost; what kind of loss that removal is becomes visible only when the hit rate and the unlabeled share are read together. #### Summary - The prompt is split into three components: **instruction** (what to do), **context** (in which world), **input** (which record). On 120 fictional tickets the shares are 0.3945 / 0.3114 / 0.2941; the real data is about a third of the prompt. - The context closes the answer space: when removed, the hit rate goes from 0.7833 to 0.7083, the unlabeled response share from 0.0250 to 0.0667, and 1797 tokens are gained. - The instruction is the longest and the best-return component: when removed, 2280 tokens are gained on input but 1057 tokens are paid back on output, and the unlabeled share rises to 0.2167. - A prompt without input also produces a well-formed response and spends 4197 tokens; its hit rate is 0.1417. A well-formed output does not mean that the prompt has been measured. - The two-run difference of the full prompt is 0.0583; of the prompts missing the context or the instruction, between 0.2000 and 0.2583. The measured contribution of the components is as much **narrowing the movement between runs** as raising the hit rate. #### Next Step In this lesson the instruction and the context were rewritten into each of one hundred twenty requests and held 0.7059 of the total input. Because the code did this in a single line, the cost stayed invisible: the same thirty-four tokens were sent one hundred twenty times. A behavior definition is written once but **paid for in every request**, and as the number of requests grows this repetition becomes the dominant line item of the total cost. The next lesson measures this: how the total tokens of a persistent system instruction grow with the number of requests, how many tokens shortening the instruction saves, and how much of the hit rate that shortening gives back. ### System Instruction Source: https://academia.sh/en/courses/prompt-engineering/system-instruction Measuring how a persistent behavior definition is sent again in every request and how its cost is multiplied by the number of requests: a 62-token instruction holds 0.8378 of a single prompt and 7440 tokens across 120 requests. Four lengths are set side by side. The prompt without an instruction gives a label hit rate of 0.4583 and 2985 tokens, the short instruction 0.7167 and 4751, the medium instruction 0.7500 and 5856, the long instruction 0.7417 and 9343. Cutting the long instruction to the medium one saves 3487 tokens and gives up nothing in hit rate; cutting the medium one to the short one saves 1105 tokens but raises the unlabeled response share from 0.0250 to 0.2167. The previous lesson split the prompt into three components and measured that the instruction and the context held 0.7059 of the total input. That measurement was made inside a single request. When the code sent the same two texts one hundred twenty times, the cost stayed hidden in a single line, because the repetition was inside the loop and was not counted anywhere. This lesson counts that repetition. A **system instruction** is the persistent behavior definition written so that the model behaves the same way in every request: what the task is, which options exist, in which form the response is given, what to do under ambiguity. Its name is persistent but the place where it is stored is the prompt itself; the model remembers nothing between two calls. The instruction is therefore **written once and paid for in every request**. That is what will be measured: the total of the repetition, the share of the repetition, and how much of the hit rate shortening the instruction gives back. - **PD9.** The code calls a **simulator**; no real model endpoint is called. The support tickets are **fictional** and are the same 120 records as in the previous lesson. - **PD10.** The system instruction and the input are inside the same prompt. The simulator has no separate instruction field, because what is measured is the **token cost** and the name of the field does not change the cost. - **PD11.** Four instruction lengths are built incrementally: none, short (the label set only), medium (the label set plus the task and the response form), long (that, with ambiguity rules added). - **PD12.** The cost of the instruction is **length times number of requests**. In the table the `repeat` column is this product and it is included in the total input tokens. - **PD13.** The criterion is the label hit rate; the unlabeled response share is also printed. Tokens per correct label is found by dividing total tokens by the hit rate and the record count. - **PD14.** Every row is run with two seeds (20260218 and 20260219). A hit rate difference that stays smaller than the two-run difference counts as unmeasured in this course. - **PD15.** The rate limit fires on every thirty-seventh call and returns an empty response. In one hundred twenty calls three records are left unlabeled because of it; this share belongs not to the prompt but to **the surface**. - **PD16.** Token cost and budget were measured in the closing lesson of the previous course; here only the token unit is used. #### The Instruction Does Not Stay in the Model The first block sets up the simulator and the fiction, defines the four instruction lengths incrementally and writes the total cost of each over one hundred twenty requests. At the end a single fictional ticket is called with the short and the medium instruction, with two separate seeds. ```python # SIMULATOR. This code does not call a real model endpoint; it is a language model # simulator written with the standard library. The support tickets are FICTIONAL too. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUT = 240, 24 # context window and maximum output, in tokens def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer: pieces of at most five characters b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def prompt_trace(m): # trace that depends on the prompt content, independent of the run h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] TEMPLATES = { "fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"], } EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim", "acil bilgi almam gerekiyor", "kayit numaram elimde yok"] TICKETS = [] for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = TEMPLATES[k][int(r() * 3)] if r() < 0.34: m += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": m, "topic": k}) CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "borc": "odeme", "odedi": "odeme", "ucret": "odeme", "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet", "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"} TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} # misleading cues FREE_FORM = ["talebiniz {k} konusuyla ilgili gorunuyor", "bu kayit {k} basligina girer , ilgili ekibe yonlendirildi", "kaydiniz alindi , konuyu inceleyip donus yapacagiz"] CALL_COUNT = [0] def call(prompt, seed=SEED, temperature=0.6, max_out=MAX_OUT): CALL_COUNT[0] += 1 tok = tokenize(prompt) if len(tok) > WINDOW: return {"status": "context_overflow", "tokens_in": len(tok), "tokens_out": 0, "response": ""} if CALL_COUNT[0] % 37 == 0: return {"status": "rate_limit", "tokens_in": len(tok), "tokens_out": 0, "response": ""} r = rng(seed ^ prompt_trace(prompt)) if len(tok) > 90 and r() < 0.04: return {"status": "timeout", "tokens_in": len(tok), "tokens_out": 0, "response": ""} cand = [k for k in CONCEPTS if k[:5] in tok] # concepts named in the prompt if len(cand) < 3: cand = CONCEPTS p = {k: 0.15 for k in cand} for t in tok: for table, w in ((CUES, 1.0), (TRAPS, 1.3)): if table.get(t) in p: p[table[t]] += w if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) # most likely choice else: e = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [v / sum(e) for v in e])] y = k if "tek" in tok else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k) ot = tokenize(y) if len(ot) > max_out: # stop condition ot = ot[:max_out] y = " ".join(ot) return {"status": "ok", "tokens_in": len(tok), "tokens_out": len(ot), "response": y} def read_label(y): # extract a label from the response for s in y.lower().split(): if s in TOPICS: return s return None E1 = "Etiketler: fatura , ariza , sayac , tarife , kesinti ." E2 = " Talebi bu etiketlerden birine ata . Yanit olarak tek etiket yaz ." E3 = (" Birden cok etiket uyuyorsa talebin ana konusunu sec ." " Hicbiri uymuyorsa en yakin etiketi sec . Aciklama ekleme .") INSTRUCTIONS = (("none", ""), ("short", E1), ("medium", E1 + E2), ("long", E1 + E2 + E3)) def build_prompt(s, x): # persistent instruction + input that changes per request return (s + " " if s else "") + "Talep: " + x["text"] print(f"{'instruction':<11} {'tokens':>9} {'in 120 requests':>15}") for name, s in INSTRUCTIONS: b = len(tokenize(s)) print(f"{name:<11} {b:>9d} {b * len(TICKETS):>15d}") x0 = TICKETS[1] gb = len(tokenize("Talep: " + x0["text"])) ub = len(tokenize(INSTRUCTIONS[3][1])) print(f"long prompt: instruction {ub} + input {gb} tokens, instruction share " f"{ub / (ub + gb):.4f}") print(f"FICTIONAL ticket {x0['id']} ({x0['topic']}): {x0['text']}") for name, s in INSTRUCTIONS[1:3]: CALL_COUNT[0] = 0 for sd in (SEED, SEED + 1): print(f"{name} / seed {sd}: {call(build_prompt(s, x0), seed=sd)['response']}") ``` ``` instruction tokens in 120 requests none 0 0 short 15 1800 medium 33 3960 long 62 7440 long prompt: instruction 62 + input 12 tokens, instruction share 0.8378 FICTIONAL ticket 4002 (fatura): fatura ayrintisini nereden gorebilirim short / seed 20260218: talebiniz fatura konusuyla ilgili gorunuyor short / seed 20260219: bu kayit fatura basligina girer , ilgili ekibe yonlendirildi medium / seed 20260218: fatura medium / seed 20260219: fatura ``` The last four lines show what the medium instruction does. With the short instruction the two runs give **two different sentences**, and the second one contains no label at all; with the medium instruction both runs give a single word, and the label is correct in both. What changes is not the hit rate but the **readability** of the response, and its cost is eighteen tokens. The lines above are the actual proportion to be measured. The long instruction is 62 tokens; the ticket to be classified is 12 tokens. In a single prompt the instruction holds a **share of 0.8378**. Across one hundred twenty requests the same 62 tokens are sent one hundred twenty times and the total rises to 7440 tokens. Dropping one token from the instruction saves one hundred twenty tokens over one hundred twenty requests; this means that every decision taken while writing the instruction is **multiplied by the number of requests**. #### The Total of the Repetition The second block runs the four instructions on the same 120 records, then repeats the long instruction with three different request counts, and finally sets the tokens that each shortening gains next to the hit rate it drops. ```python def measure(s, seed=SEED, n=None): CALL_COUNT[0] = 0 dataset = TICKETS if n is None else TICKETS[:n] ti = to = d = unl = 0 for x in dataset: c = call(build_prompt(s, x), seed=seed) ti, to = ti + c["tokens_in"], to + c["tokens_out"] e = read_label(c["response"]) unl += e is None d += e == x["topic"] m = len(dataset) return {"hit_rate": d / m, "unlabeled": unl / m, "tokens_in": ti, "tokens_out": to, "instruction": len(tokenize(s)) * m, "total": ti + to} R = {name: measure(s) for name, s in INSTRUCTIONS} print(f"{'instruction':<11} {'repeat':>7} {'in':>6} {'share':>7} {'out':>6} {'hit rate':>8} " f"{'run 2':>8} {'unlabeled':>10} {'per correct':>13}") for name, s in INSTRUCTIONS: v, k2 = R[name], measure(s, seed=SEED + 1)["hit_rate"] print(f"{name:<11} {v['instruction']:>7d} {v['tokens_in']:>6d} " f"{v['instruction'] / v['tokens_in']:>7.4f} " f"{v['tokens_out']:>6d} {v['hit_rate']:>8.4f} {k2:>8.4f} {v['unlabeled']:>10.4f} " f"{v['total'] / v['hit_rate'] / len(TICKETS):>13.1f}") print() print(f"{'requests':>8} {'instruction tokens':>18} {'total input':>13} {'share':>7}") for n in (30, 60, 120): v = measure(INSTRUCTIONS[3][1], n=n) print(f"{n:>8d} {v['instruction']:>18d} {v['tokens_in']:>13d} " f"{v['instruction'] / v['tokens_in']:>7.4f}") print() print(f"{'shortening':<16} {'tokens gained':>19} {'hit rate diff':>13} " f"{'per 0.01 hit rate':>19}") for (a1, _), (a2, _) in zip(INSTRUCTIONS[1:], INSTRUCTIONS[:-1]): kb = R[a1]["total"] - R[a2]["total"] fi = R[a2]["hit_rate"] - R[a1]["hit_rate"] o = f"{kb * 0.01 / abs(fi):.0f} tokens" if fi < 0 else "lossless" print(f"{a1 + ' -> ' + a2:<16} {kb:>19d} {fi:>+13.4f} {o:>19}") ``` ``` instruction repeat in share out hit rate run 2 unlabeled per correct none 0 1700 0.0000 1285 0.4583 0.6333 0.2917 54.3 short 1800 3500 0.5143 1251 0.7167 0.4583 0.2167 55.2 medium 3960 5660 0.6996 196 0.7500 0.7500 0.0250 65.1 long 7440 9140 0.8140 203 0.7417 0.7583 0.0250 105.0 requests instruction tokens total input share 30 1860 2295 0.8105 60 3720 4547 0.8181 120 7440 9140 0.8140 shortening tokens gained hit rate diff per 0.01 hit rate short -> none 1766 -0.2583 68 tokens medium -> short 1105 -0.0333 332 tokens long -> medium 3487 +0.0083 lossless ``` The second table shows how the repetition behaves. At thirty requests the long instruction comes to 1860, at sixty requests to 3720, at one hundred twenty requests to 7440 tokens: the cost grows **linearly** with the number of requests. The share of the instruction in total input stays steady between 0.8105 and 0.8181. There is no such thing as the share of the repetition thinning out as scale grows; whatever the number of requests, more than three quarters of the input is the instruction. However well an instruction is written, its cost is **not its own length but its length multiplied by traffic**. #### The Cost of Shortening The first table says where the hit rate saturates. The prompt without an instruction gives 0.4583. The short instruction raises it to 0.7167. The medium instruction raises it to 0.7500. The long instruction gives 0.7417, that is, **below the medium one**. The twenty-nine-token addition — ambiguity rules, main topic selection, the ban on explanation — added a total of 3487 tokens and took 0.0083 away from the hit rate. The cost per correct label went from 65.1 to 105.0 tokens. In this course such a row is written by name: **the cost of the added text is certain, its gain is uncertain.** The third table reads the shortening from the opposite direction. Going from the long instruction to the medium one saves 3487 tokens and takes nothing from the hit rate; this is a **lossless shortening**. Going from medium to short saves 1105 tokens and takes 0.0333 of hit rate, that is, 332 tokens for every 0.01 of hit rate. Going from short to none saves 1766 tokens and takes 0.2583 of hit rate, that is, 68 tokens for every 0.01 of hit rate. The return on shortening is not constant: the shorter the instruction gets, the more expensive each remaining token becomes. The `unlabeled` column of the table shows the real loss of shortening. With no instruction, no label can be read from 0.2917 of the responses, and with the short instruction none can be read from 0.2167 of them. With the medium instruction this share falls to 0.0250 and stays there. The remaining three records belong not to the prompt but to **the surface**: the rate limit returns three of the one hundred twenty calls empty. So the medium instruction has closed the gap that can be closed with a prompt. Since the long instruction has nothing left to add, it adds only tokens. Finally the `run 2` column is read. The prompt without an instruction gave 0.4583 and 0.6333 in two runs, the short instruction 0.7167 and 0.4583: in the second run **the prompt without an instruction beat the prompt with the short instruction**. The gain of the short instruction shows in one run and does not carry over to two. The medium instruction gave 0.7500 in both runs. The long instruction gave 0.7417 and 0.7583; its own two runs differ by 0.0167. The 0.0083 difference between the long and the medium instruction stays below that and **counts as unmeasured** in this course. The rule is applied by name in the next lesson. #### Summary - The system instruction is a persistent behavior definition, but its persistence is in the prompt, not in the model: it is sent again in every request and **paid for again in every request**. - The 62-token long instruction holds 0.8378 of a single prompt and 7440 tokens across 120 requests. The share is steady between 0.8105 and 0.8181 at thirty, sixty and one hundred twenty requests; the total grows linearly. - The hit rate saturates at the medium instruction: none 0.4583, short 0.7167, medium 0.7500, long 0.7417. The long instruction added 3487 tokens, took 0.0083 from the hit rate and raised the cost per correct label from 65.1 to 105.0. - The return on shortening is not constant: going from long to medium is lossless, going from medium to short gains 332 tokens per 0.01 of hit rate, going from short to none gains 68 tokens. - The real gap the instruction closes is the unlabeled response share: it falls from 0.2917 to 0.0250 and stays there. The remaining share comes from the rate limit, so it cannot be closed with a prompt. #### Next Step The difference between the medium and the long instruction stayed below the own run difference of the long one: the twenty-nine added tokens produced nothing measurable. This does not close the question of what to add to an instruction. A frequent answer is to tell the model **who it is**: an expertise, a job title, a style. That this addition raises the hit rate is a common belief, and just like the ambiguity rules it is paid for in every request. The next lesson measures this: what difference specifying a role adds, how many tokens it costs, and whether that difference is larger than the difference between two runs of the same prompt. ### Specifying Role and Behavior Source: https://academia.sh/en/courses/prompt-engineering/specifying-role-and-behavior Measuring the difference that adding a role and behavior prefix to a prompt makes and comparing it with the run difference of the same prompt. On the same 120 fictional support tickets the hit rate of the prefixed prompts moves -0.0667, -0.0583, -0.2208 and +0.0125 against the 0.7500 base, while the band of the base across four seeds is 0.1500. None of the four prefixes leaves the band, so none of the differences counts as measured. The only thing measured is the cost: prefixes of 9 to 27 tokens add 1080 to 3240 tokens over 120 requests and raise the cost per correct label from 65.1 to between 83.6 and 123.8. The previous lesson measured the length of the instruction and found that the hit rate saturates at thirty-three tokens; the twenty-nine tokens added after that raised the cost for certain and did not move the number in any measurable way. The question of what to add to the instruction was not closed there. The most frequent answer is to tell the model **who it is**: an area of expertise, a job title, a level of seniority. That this addition raises the hit rate is a common belief. This lesson measures that belief and, while measuring, puts the third rule of the course to work by name. **The model is an uncertain component**: the same prompt gives a different number when the seed changes. If the difference that a prompt change adds is not larger than the range the same prompt moves from run to run, that difference is not a gain but a coincidence. In this course such a number **counts as unmeasured**. The lesson builds this criterion, applies it to four prefixes and writes the result openly. - **PD17.** The code calls a **simulator**; no real model endpoint is called. The support tickets are **fictional** and are the same 120 records as in the previous two lessons. - **PD18.** The base prompt is the medium instruction measured in the previous lesson: the label set, the task and the response form. In this lesson the instruction does not change; only a **prefix** is added in front of it. - **PD19.** **Role prompting** (specifying a role) is text that attributes an identity to the model. **Specifying a behavior** is text that writes the steps the model is asked to follow. Both are placed in front of the instruction in the same way. - **PD20.** Every prompt is run with **four seeds** (20260218 to 20260221). The first two seeds are the **two runs** columns; the difference between the largest and the smallest of the four is the **run band**. - **PD21.** The **effect** of a prefix is the difference between the mean of its first two runs and the mean of the first two runs of the base. - **PD22.** **Measurability criterion:** an effect that is not larger than **the larger of the bands** of the two compared prompts counts as unmeasured. The criterion is conservative in this respect and its reason is written in the lesson. - **PD23.** The cost is in tokens. The prefix cost is **prefix length times number of requests** and it is included in the total tokens; the cost per correct label is printed separately. - **PD24.** The trace table of the simulator is in the code and is readable. Which label a prefix text adds weight to is verified from this table. - **PD25.** This lesson measures only the hit rate and the cost; the form of the response was closed in the previous lesson and is fixed here. #### In a Single Ticket the Role Seems to Have Worked The first block sets up the simulator and the fiction, defines the four prefixes, writes the cost of each over one hundred twenty requests and calls a single fictional ticket with the base and with the role prefix, with two separate seeds. ```python # SIMULATOR. This code does not call a real model endpoint; it is a language model # simulator written with the standard library. The support tickets are FICTIONAL too. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUT = 240, 24 # context window and maximum output, in tokens def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer: pieces of at most five characters b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def prompt_trace(m): # trace that depends on the prompt content, independent of the run h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] TEMPLATES = { "fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"], } EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim", "acil bilgi almam gerekiyor", "kayit numaram elimde yok"] TICKETS = [] for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = TEMPLATES[k][int(r() * 3)] if r() < 0.34: m += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": m, "topic": k}) CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "borc": "odeme", "odedi": "odeme", "ucret": "odeme", "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet", "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"} TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} # misleading cues FREE_FORM = ["talebiniz {k} konusuyla ilgili gorunuyor", "bu kayit {k} basligina girer , ilgili ekibe yonlendirildi", "kaydiniz alindi , konuyu inceleyip donus yapacagiz"] CALL_COUNT = [0] def call(prompt, seed=SEED, temperature=0.6, max_out=MAX_OUT): CALL_COUNT[0] += 1 tok = tokenize(prompt) if len(tok) > WINDOW: return {"status": "context_overflow", "tokens_in": len(tok), "tokens_out": 0, "response": ""} if CALL_COUNT[0] % 37 == 0: return {"status": "rate_limit", "tokens_in": len(tok), "tokens_out": 0, "response": ""} r = rng(seed ^ prompt_trace(prompt)) if len(tok) > 90 and r() < 0.04: return {"status": "timeout", "tokens_in": len(tok), "tokens_out": 0, "response": ""} cand = [k for k in CONCEPTS if k[:5] in tok] # concepts named in the prompt if len(cand) < 3: cand = CONCEPTS p = {k: 0.15 for k in cand} for t in tok: for table, w in ((CUES, 1.0), (TRAPS, 1.3)): if table.get(t) in p: p[table[t]] += w if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) # most likely choice else: e = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [v / sum(e) for v in e])] y = k if "tek" in tok else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k) ot = tokenize(y) if len(ot) > max_out: # stop condition ot = ot[:max_out] y = " ".join(ot) return {"status": "ok", "tokens_in": len(tok), "tokens_out": len(ot), "response": y} def read_label(y): # extract a label from the response for s in y.lower().split(): if s in TOPICS: return s return None INSTRUCTION = ("Etiketler: fatura , ariza , sayac , tarife , kesinti ." " Talebi bu etiketlerden birine ata . Yanit olarak tek etiket yaz .") PREFIXES = (("base", ""), ("role A", "Sen bir abone destek gorevlisisin ."), ("role B", "Sen bir deneyimli saha teknisyenisin ."), ("role C", "Sen dagitim sirketinde on yildir calisan kidemli bir kayit uzmanisin ."), ("behavior", "Once talebin ana nesnesini bul , sonra etiketi sec ." " Emin olmadigin durumda talepte gecen ana sozcugu esas al .")) SEEDS = (SEED, SEED + 1, SEED + 2, SEED + 3) def build_prompt(p, x): return (p + " " if p else "") + INSTRUCTION + " Talep: " + x["text"] print(f"{'prefix':<9} {'tokens':>9} {'in 120 requests':>15}") for name, p in PREFIXES: b = len(tokenize(p)) print(f"{name:<9} {b:>9d} {b * len(TICKETS):>15d}") x0 = TICKETS[2] print(f"FICTIONAL ticket {x0['id']} ({x0['topic']}): {x0['text']}") for name, p in PREFIXES[:2]: CALL_COUNT[0] = 0 y = [call(build_prompt(p, x0), seed=t)["response"] for t in SEEDS[:2]] print(f"{name:<9} two runs: {y[0]} / {y[1]}") ``` ``` prefix tokens in 120 requests base 0 0 role A 9 1080 role B 9 1080 role C 17 2040 behavior 27 3240 FICTIONAL ticket 4003 (ariza): evde su akmiyor bir ariza var galiba ayrica gecen ay da ayni sorun olmustu base two runs: kesinti / ariza role A two runs: ariza / ariza ``` The last two lines show how a prompt change comes to look proven. The real topic of the ticket is `fault`. The base prompt answered wrong once and right once in two runs; when the role prefix was added, the answer came right in both runs. Anyone looking at the single ticket would conclude "the role raised the hit rate". The only ground for this conclusion is that the base **already moves from run to run**. The same demonstration could just as well have been made with a ticket that came out in the opposite direction. #### What Happens Across the Dataset The second block runs the same four prefixes on all 120 records and with four separate seeds. In every row the first two runs, the difference of their mean from the base, the band of the four seeds and the cost per correct label stand side by side. ```python def measure(p, seed): CALL_COUNT[0] = 0 ti = to = d = 0 for x in TICKETS: c = call(build_prompt(p, x), seed=seed) ti, to = ti + c["tokens_in"], to + c["tokens_out"] d += read_label(c["response"]) == x["topic"] return {"hit_rate": d / len(TICKETS), "total": ti + to} S = {name: [measure(p, t) for t in SEEDS] for name, p in PREFIXES} bh = [v["hit_rate"] for v in S["base"]] base_mean, base_band = (bh[0] + bh[1]) / 2, max(bh) - min(bh) print(f"{'prompt':<9} {'prefix':>6} {'total':>7} {'run 1':>8} {'run 2':>8} {'effect':>8} " f"{'two runs':>10} {'band':>7} {'measurable':>10} {'per correct':>11}") for name, p in PREFIXES: h = [v["hit_rate"] for v in S[name]] mean = (h[0] + h[1]) / 2 effect, gap, band = mean - base_mean, abs(h[0] - h[1]), max(h) - min(h) total = S[name][0]["total"] limit = max(band, base_band) print(f"{name:<9} {len(tokenize(p)):>6d} {total:>7d} {h[0]:>8.4f} {h[1]:>8.4f} {effect:>+8.4f} " f"{gap:>10.4f} {band:>7.4f} {('yes' if abs(effect) > limit else 'no'):>10} " f"{total / mean / len(TICKETS):>11.1f}") ``` ``` prompt prefix total run 1 run 2 effect two runs band measurable per correct base 0 5856 0.7500 0.7500 +0.0000 0.0000 0.1500 no 65.1 role A 9 6940 0.7083 0.6583 -0.0667 0.0500 0.0667 no 84.6 role B 9 6942 0.7250 0.6583 -0.0583 0.0667 0.1417 no 83.6 role C 17 7863 0.6417 0.4167 -0.2208 0.2250 0.2250 no 123.8 behavior 27 9105 0.7083 0.8167 +0.0125 0.1083 0.1083 no 99.5 ``` #### Two Runs Are the Lower Bound of the Band The first row of the table shows why the criterion was built in this form. The base prompt gave exactly 0.7500 in both runs; the two-run difference is **0.0000**. Looking at this number one could have said "this prompt is stable". When four seeds are run the base gives 0.7500, 0.7500, 0.8583 and 0.7083 and its band comes out at **0.1500**. The two runs coinciding is not the absence of uncertainty but two samples falling on the same spot. **Two runs are not the measure of the band but its lower bound.** The criterion was made conservative for this reason: an effect that is not larger than the larger of the bands of the two compared prompts counts as unmeasured. None of the four prefixes passes this threshold. The effect of role A is −0.0667 and its threshold 0.1500. The effect of role B is −0.0583 and its threshold 0.1500. The effect of the behavior prefix is +0.0125 and its threshold 0.1500. The effect of role C is −0.2208 and its threshold is its own band, 0.2250. With a looser reading, if only the band of the base had been looked at, role C would pass the threshold, but the direction in which it passes would be **down**. Whichever reading is chosen the result is the same: **none of the four prefixes raised the hit rate in a measurable way.** The only thing measured is the cost, and the cost carries no band. Role A and role B are nine tokens each; 1080 tokens over one hundred twenty requests, and a rise from 5856 to 6940 and 6942 in total. Role C is seventeen tokens, 2040 repeated, total 7863. The behavior prefix is twenty-seven tokens, 3240 repeated, total 9105. The cost per correct label rises from 65.1 tokens to 84.6, 83.6, 123.8 and 99.5 tokens. The prompt that is most expensive per correct label is **1.90 times** the base, and it gives nothing measurable in return. #### The Role Text Is Not Neutral The effect of the role prefixes came out unmeasurable, but the **direction** of their effects is not random. The trace table of the simulator is in the code and is readable. The `abone` (subscriber) fragment in the text of role A is tied to the `tariff` label in the trace table; the `calis` (work) fragment in the text of role C is tied to `fault`, and the `kayit` (record) fragment is again tied to `fault` in the table of misleading cues. So role C adds weight to the `fault` label in every ticket, independently of the content of the ticket. No fragment in the text of role B appears in either table; its effect is only run noise. The structural conclusion that follows holds for every text written into a prompt. **The role text is not a separate channel**; it enters the same token sequence as the task text, and the words it carries mix into the decision. Which words are chosen for a role determines more than how convincing that role is written. While the prompt writer sees the role sentence as style, the model reads it as evidence. This does not mean that specifying a role is ineffective for every task; this measurement was made on a single task, a single dataset and a single simulator. What it means is narrower and continues throughout the course: **a prompt with a role added is not an improved prompt until the difference is measured.** Its cost, on the other hand, is known without any measurement, because it is paid for again in every request. #### Summary - In a single ticket the role prefix corrected the answer the base got wrong. The same prefix brought the hit rate over 120 tickets from 0.7500 down to 0.6833. A demonstration over a single ticket presents the movement of the base itself as evidence. - The base prompt gave 0.7500 in both runs, a two-run difference of 0.0000. Its band across four seeds is **0.1500**. Two runs are not the measure of the band but its **lower bound**. - Measurability criterion: an effect that is not larger than the larger of the bands of the two compared prompts counts as unmeasured. Role A −0.0667, role B −0.0583, role C −0.2208, behavior +0.0125; **all four are below the threshold**. - The cost carries no band and is fully measured: prefixes of 9 to 27 tokens, 1080 to 3240 tokens over 120 requests. The cost per correct label rises from 65.1 to 84.6, 83.6, 123.8 and 99.5 tokens. - The role text is not a separate channel. The `abone` fragment is tied to `tariff` in the trace table, the `calis` and `kayit` fragments to `fault`; the role sentence is read in the same sequence as the task text and mixes into the decision. #### Next Step In this lesson an identity and a procedure were added to the prompt, but **what was asked** did not change at all: the request was the same sentence in all three prompts. The task definition itself is a design decision, and when it is left ambiguous its cost is larger than that of the prefixes. An ambiguous request does not produce a single wrong answer; it produces **a distribution**, and that distribution is not visible in a single run. The next lesson measures this: in how many different forms the same ambiguous prompt is answered across ten runs, how far that number falls when the request is made measurable, and how many tokens the narrowing of the distribution costs. ### Clear and Measurable Requests Source: https://academia.sh/en/courses/prompt-engineering/clear-and-measurable-requests Measuring that an ambiguous request produces not a single wrong response but a distribution: the same ambiguous prompt gives a mean of 4.81 distinct responses per record in ten runs, the most frequent response is repeated in only 0.3944 of the runs, and no label can be read from 0.4333 of the responses. Four request forms are set side by side. Clarifying the verb adds two tokens and lowers the distribution to 4.53; closing the option set adds 15 tokens and lowers the distribution to 3.89 and the unlabeled share to 0.2889; adding an ambiguity rule adds 10 more tokens and raises the unlabeled share back to 0.3722. The band of the hit rate read in a single run is between 0.2778 and 0.3333 and cannot tell the four requests apart. The previous lesson added an identity and a procedure to the prompt and found that the difference of none of the four prefixes is measurable. Throughout that measurement one thing never changed: **what was asked**. The request was the same sentence in every prompt, and that sentence itself was never questioned. Yet the task definition is not the most expensive part of the prompt but its most decisive part. The cost of an ambiguous request is not that it gives an ambiguous response. An ambiguous request does not produce a single wrong answer; it produces **a distribution**. The same prompt, the same record, the same model, and in ten separate runs a few separate responses that do not replace one another. This distribution is not visible in any measure that looks at a single run; what shows in a single run is one sample drawn from that distribution. This lesson measures the distribution and counts how far making the request measurable narrows it. - **PD26.** The code calls a **simulator**; no real model endpoint is called. The support tickets are **fictional** and are the first 36 records of the dataset in the previous lessons. The dataset was made smaller because every record is called ten times. - **PD27.** Every record is measured with **ten runs**: seeds 20260218 to 20260227. The distribution measures are the total of these ten responses, not of a single run. - **PD28.** The **number of distinct responses** is the number of response texts that come out different from one another in ten runs, averaged per record. Its smallest value is 1 and its largest is 10. - **PD29.** The **top share** is the share of the most repeated response in ten runs. It approaches 1 in a narrow distribution and 0.1 in a wide one. - **PD30.** The rate limit fires on every thirty-seventh call and returns an empty response. So that it does not disturb the distribution count, the call counter is reset before the ten runs of every record; this is a measurement decision and it is in the code. - **PD31.** The four request forms are built incrementally: a vague verb, a clear verb, a clear verb with a closed option set, and that with an ambiguity rule added. - **PD32.** In this lesson **the form of the response is not requested in any prompt**. What is measured is the clarity of the request; the form of the output is the subject of the next lesson. - **PD33.** The hit rate is computed separately in each of the ten runs; the table prints its mean and its **band** (the difference between the largest and the smallest run). - **PD34.** The cost is in tokens and is written per request as input and output; the cost of ten runs is ten times that. #### The Distribution an Ambiguous Request Produces The first block sets up the simulator and the fiction, defines the four request forms, calls a single fictional record ten times with the most ambiguous form and counts the responses that come out. ```python # SIMULATOR. This code does not call a real model endpoint; it is a language model # simulator written with the standard library. The support tickets are FICTIONAL too. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUT = 240, 24 # context window and maximum output, in tokens def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer: pieces of at most five characters b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def prompt_trace(m): # trace that depends on the prompt content, independent of the run h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] TEMPLATES = { "fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"], } EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim", "acil bilgi almam gerekiyor", "kayit numaram elimde yok"] TICKETS = [] for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = TEMPLATES[k][int(r() * 3)] if r() < 0.34: m += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": m, "topic": k}) CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "borc": "odeme", "odedi": "odeme", "ucret": "odeme", "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet", "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"} TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} # misleading cues FREE_FORM = ["talebiniz {k} konusuyla ilgili gorunuyor", "bu kayit {k} basligina girer , ilgili ekibe yonlendirildi", "kaydiniz alindi , konuyu inceleyip donus yapacagiz"] CALL_COUNT = [0] def call(prompt, seed=SEED, temperature=0.6, max_out=MAX_OUT): CALL_COUNT[0] += 1 tok = tokenize(prompt) if len(tok) > WINDOW: return {"status": "context_overflow", "tokens_in": len(tok), "tokens_out": 0, "response": ""} if CALL_COUNT[0] % 37 == 0: return {"status": "rate_limit", "tokens_in": len(tok), "tokens_out": 0, "response": ""} r = rng(seed ^ prompt_trace(prompt)) if len(tok) > 90 and r() < 0.04: return {"status": "timeout", "tokens_in": len(tok), "tokens_out": 0, "response": ""} cand = [k for k in CONCEPTS if k[:5] in tok] # concepts named in the prompt if len(cand) < 3: cand = CONCEPTS p = {k: 0.15 for k in cand} for t in tok: for table, w in ((CUES, 1.0), (TRAPS, 1.3)): if table.get(t) in p: p[table[t]] += w if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) # most likely choice else: e = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [v / sum(e) for v in e])] y = k if "tek" in tok else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k) ot = tokenize(y) if len(ot) > max_out: # stop condition ot = ot[:max_out] y = " ".join(ot) return {"status": "ok", "tokens_in": len(tok), "tokens_out": len(ot), "response": y} def read_label(y): # extract a label from the response for s in y.lower().split(): if s in TOPICS: return s return None VERB = "Su destek talebini konusuna gore etiketle ." SET_TEXT = " Etiketler: fatura , ariza , sayac , tarife , kesinti ." REQUEST_FORMS = (("vague verb", "Su destek talebini degerlendir ."), ("clear verb", VERB), ("closed set", VERB + SET_TEXT), ("ambiguity rule", VERB + SET_TEXT + " Birden cok konu varsa ana konuyu sec .")) DATASET = TICKETS[:36] SEEDS = [SEED + i for i in range(10)] def runs(head, x): CALL_COUNT[0] = 0 # reset for every ticket so the rate limit does not disturb the distribution count return [call(head + " Talep: " + x["text"], seed=t) for t in SEEDS] def breakdown(name, head, x): s = [c["response"] for c in runs(head, x)] print(f"{name}, ten runs -> {len(set(s))} distinct responses") for v in sorted(set(s)): print(f" {s.count(v)}/10 {v}") print(f"{'request':<19} {'tokens':>9}") for name, h in REQUEST_FORMS: print(f"{name:<19} {len(tokenize(h)):>9d}") x0 = DATASET[3] print(f"FICTIONAL ticket {x0['id']} (true topic {x0['topic']}): {x0['text']}") breakdown(REQUEST_FORMS[0][0], REQUEST_FORMS[0][1], x0) ``` ``` request tokens vague verb 9 clear verb 11 closed set 26 ambiguity rule 36 FICTIONAL ticket 4004 (true topic ariza): ariza kaydi actim ama ekip gelmedi vague verb, ten runs -> 7 distinct responses 1/10 bu kayit ariza basligina girer , ilgili ekibe yonlendirildi 1/10 bu kayit odeme basligina girer , ilgili ekibe yonlendirildi 1/10 bu kayit sayac basligina girer , ilgili ekibe yonlendirildi 2/10 kaydiniz alindi , konuyu inceleyip donus yapacagiz 1/10 talebiniz fatura konusuyla ilgili gorunuyor 2/10 talebiniz kesinti konusuyla ilgili gorunuyor 2/10 talebiniz sikayet konusuyla ilgili gorunuyor ``` A single record gave **seven distinct responses** in ten runs. The most frequent response was repeated in only two runs. One of the responses carries no label at all; two others use the labels `odeme` (payment) and `sikayet` (complaint), that is, names that have no counterpart in the fictional document vocabulary of the field team. The prompt said "evaluate", and that verb never said which set a result was wanted from or in what form. The model filled this gap in a different way across the runs. The real meaning of this table lies not in one record but in the distribution across records. An observer who looks at only one of the seven distinct responses reaches the conclusion that the prompt worked or did not work with equal ease. #### Four Requests Side by Side The second block measures the four request forms on thirty-six records and with ten runs. In every row stand the mean number of distinct responses per record, the share of the most frequent response, the unlabeled response share, the mean hit rate of the ten runs, the band of the hit rate and the token cost. ```python def measure(head): dr = tp = unl = ti = to = 0 hits = [0] * len(SEEDS) for x in DATASET: c = runs(head, x) s = [v["response"] for v in c] e = [read_label(v) for v in s] dr += len(set(s)) tp += max(s.count(v) for v in s) / len(s) unl += sum(v is None for v in e) / len(e) for j, v in enumerate(e): hits[j] += v == x["topic"] ti += sum(v["tokens_in"] for v in c) / len(c) to += sum(v["tokens_out"] for v in c) / len(c) n = len(DATASET) hr = [v / n for v in hits] return (dr / n, tp / n, unl / n, sum(hr) / len(hr), max(hr) - min(hr), ti / n, to / n) print(f"{'request':<19} {'tok':>4} {'distinct':>11} {'top share':>9} {'unlabeled':>10} " f"{'hit rate':>8} {'band':>7} {'in':>6} {'out':>6}") for name, h in REQUEST_FORMS: dr, tp, unl, hr, band, ti, to = measure(h) print(f"{name:<19} {len(tokenize(h)):>4d} {dr:>11.2f} {tp:>9.4f} {unl:>10.4f} " f"{hr:>8.4f} {band:>7.4f} {ti:>6.1f} {to:>6.1f}") print() breakdown(REQUEST_FORMS[2][0], REQUEST_FORMS[2][1], x0) ``` ``` request tok distinct top share unlabeled hit rate band in out vague verb 9 4.81 0.3944 0.4333 0.4611 0.3333 23.1 11.4 clear verb 11 4.53 0.4333 0.4500 0.4417 0.3333 25.1 11.3 closed set 26 3.89 0.4083 0.2889 0.5306 0.2778 40.1 11.5 ambiguity rule 36 3.78 0.4611 0.3722 0.4722 0.3333 50.1 11.6 closed set, ten runs -> 4 distinct responses 2/10 bu kayit kesinti basligina girer , ilgili ekibe yonlendirildi 3/10 kaydiniz alindi , konuyu inceleyip donus yapacagiz 2/10 talebiniz ariza konusuyla ilgili gorunuyor 3/10 talebiniz kesinti konusuyla ilgili gorunuyor ``` #### Clarifying the Verb Is Not Measurability The second row is the result of this lesson that is the easiest to misunderstand. The verb "evaluate" was replaced with "label by its topic"; the request is clear as grammar and explicit as an action. The number of distinct responses fell from 4.81 to 4.53, the hit rate **fell** from 0.4611 to 0.4417, and the unlabeled response share **rose** from 0.4333 to 0.4500. The two-token addition did not narrow the distribution in any measurable way. The reason is structural. A clear verb says what to do; it does not say **which set to choose from**. A prompt that says "label" leaves it to the model where the label will come from, and in every run the model produces a name from its own vocabulary. For a request to be measurable, the verb does not need to be sharp; **the set from which the answer can be taken needs to be closed**. The third row shows this. When the names of the five labels are written into the prompt, the number of distinct responses falls to 3.89, the unlabeled share drops from 0.4333 to 0.2889 and the hit rate rises to 0.5306. The cost is plainly countable: the input per request rises from 25.1 to 40.1 tokens, that is, 15 tokens. Over thirty-six records that is 540 and over ten runs 5400 tokens. The ten runs of the same record with the closed set are at the end of the block output: seven distinct responses have fallen to four and the names outside the set have disappeared. On this record the concentration happened on the wrong label; **narrowing of the distribution is not a guarantee of hit rate**, and the two numbers must stand in separate columns. The fourth row pays off the second claim of the course. The sentence "if many topics, pick the main one" adds ten tokens and lowers the number of distinct responses from 3.89 to 3.78. The same sentence raises the unlabeled share from 0.2889 to **0.3722** and drops the hit rate from 0.5306 to **0.4722**. A sentence written to reduce ambiguity may, when measured, have increased it. #### The Distribution Is Not Visible in a Single Run The `band` column of the table is the methodological result of this lesson. The hit rate bands of the four requests across ten runs are 0.3333, 0.3333, 0.2778 and 0.3333 respectively. The effect of the closed set on the hit rate is +0.0695; exactly a quarter of its own band. Read by the criterion of the previous lesson, this difference **counts as unmeasured**. That is, which of the four requests is better cannot be told apart **in any way** with the hit rate read in a single run. The numbers that can tell them apart are different, and all of them are totals over ten runs: the number of distinct responses falls from 4.81 to 3.89 and the unlabeled share from 0.4333 to 0.2889. These two measures cannot be computed from a single run, because in a single run every record has one response and it is not visible which of how many possibilities that response is. **The cost of an ambiguous request is a cost that cannot be measured in a single run**; this is why most prompt trials never see it and close with a single successful sample. #### Summary - An ambiguous request produces not a single wrong response but a distribution. A single record gave seven distinct responses in ten runs; across the dataset the result was a mean of **4.81 distinct responses** per record and a top share of only **0.3944**. - Clarifying the verb is not measurability: the two-token addition lowered the distinct responses to 4.53, dropped the hit rate to 0.4417 and raised the unlabeled share to 0.4500. - The decision that makes the request measurable is **closing the option set**: 3.89 distinct responses, an unlabeled share of 0.2889, a hit rate of 0.5306. Its cost is 15 tokens per request, 5400 tokens over ten runs. - Adding an ambiguity rule added ten tokens and took the unlabeled share to 0.3722 and the hit rate to 0.4722: the sentence written to reduce ambiguity increased it when measured. - The band of the hit rate across ten runs is between 0.2778 and 0.3333; the +0.0695 effect of the closed set is inside the band. The distribution measures cannot be computed from a single run. #### Next Step In four lessons the components of the prompt were separated, the repetition cost of the persistent instruction was counted, the difference of the role prefixes was found unmeasurable and the request was tied to a closed set. Even so, the prompt with the closed set still produces a mean of 3.89 distinct responses per record, and no label can be read from 0.2889 of the responses. The source of the remaining distribution is no longer what is asked but **how it is said**: the same label is given in three separate sentence patterns, and sometimes with no label name at all. The request was clarified, but the form of the response is still left to the model. The next lesson takes this up: when the output form is requested, how many responses can be parsed by a machine, and how much of the hit rate the length limit takes back. ### Format and Length Control Source: https://academia.sh/en/courses/prompt-engineering/format-and-length-control The response to a clear request is still free text, and a program cannot read it. On the same 120 fictional support tickets a three-field record is requested: the share that the strict field parser reads in the free response is 0.0000, it becomes 0.9750 when a format declaration is added, and the label hit rate rises from 0.3083 to 0.6833. The cost is a rise from 6140 to 8780 in input tokens and from 1392 to 2026 in output tokens. When the maximum output length is lowered from 24 to 10, the output tokens fall from 2026 to 1170 but the parseable share drops from 0.9750 to 0.0000; at the intermediate values the curve stays tied exactly to the number of cut responses. The previous lesson counted in how many different forms an ambiguous request is answered and showed how far making the request measurable narrows this distribution. The request is now clear: the label set to choose from is written in the prompt, and what is asked stands in a single sentence. Even so, one decision is still left to the model. The model knows **what to say** but does not know **in which form** to write it. The cost of this distinction depends on who will read the response. A human reads the sentence "your record has been received, we will review the matter and get back to you" without trouble. A program that will write the same sentence into a record field cannot read it: it contains no label field, no fault code and no separate rationale. This lesson treats the form of the response as a design decision and measures two things: the difference that a format declaration adds to **machine parseability**, and the share that a length limit takes back from the same measure. - **PD41.** The model is a **simulator**; no real endpoint is called. It is written with the standard library, its seed is **20260218**, and the same prompt gives the same response with the same seed. - **PD42.** The support tickets are **fictional**: 120 short records on five topics, with a misleading sentence added to some of them. - **PD43.** The task asks for a three-field **record**: label, fault code and short rationale. The fault codes are fictional too. - **PD44.** Two parsers are used. The **strict field parser** looks for all three fields to be complete and consistent; the **loose reading** looks for a label word anywhere in the response. - **PD45.** In all three prompts the label set is the same and the request is equally clear. **The only thing that changes is the format declaration**; the length limit is measured separately. - **PD46.** The length limit is the **maximum output length** of the simulator. When the limit is exceeded the stop condition fires and the response is cut in the middle. - **PD47.** The cost is written in **tokens**: input tokens, output tokens and number of calls. #### A Free Response Is a Response the Program Cannot Read The first block sets up the simulator, defines the three-field record and writes the two parsers. Three prompts are run side by side: one without a format declaration, one with a format declaration, and one with a format declaration whose output length is limited to twelve tokens. ```python # SIMULATOR -- FICTIONAL. This code does not call a real model endpoint; it is a # language model simulator written with the standard library. The tickets are fictional too. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUT = 240, 24 # context window and maximum output, in tokens def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer: pieces of at most five characters b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def prompt_trace(m): # trace that depends on the prompt content, independent of the run h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] CODES = {"fatura": "ft-21", "ariza": "ar-14", "sayac": "sy-32", "tarife": "tr-05", "kesinti": "ks-40", "odeme": "od-11", "sikayet": "sk-09", "bilgi": "bg-02", "genel": "gn-00"} TEMPLATES = {"fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"]} EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim", "acil bilgi almam gerekiyor", "kayit numaram elimde yok"] TICKETS = [] for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = TEMPLATES[k][int(r() * 3)] if r() < 0.34: m += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": m, "topic": k}) CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "borc": "odeme", "odedi": "odeme", "ucret": "odeme", "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet", "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"} TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} # misleading cues FREE_FORM = ["talebiniz {k} konusuyla ilgili gorunuyor", "bu kayit {k} basligina girer , ilgili ekibe yonlendirildi", "kaydiniz alindi , konuyu inceleyip donus yapacagiz"] CALL_COUNT = [0] def call(prompt, summary, seed=SEED, temperature=0.6, max_out=MAX_OUT): CALL_COUNT[0] += 1 tok = tokenize(prompt) empty = {"tokens_in": len(tok), "tokens_out": 0, "response": "", "cut": 0} if len(tok) > WINDOW: return dict(empty, status="context_overflow") if CALL_COUNT[0] % 37 == 0: return dict(empty, status="rate_limit") r = rng(seed ^ prompt_trace(prompt)) if len(tok) > 90 and r() < 0.04: return dict(empty, status="timeout") cand = [k for k in CONCEPTS if k[:5] in tok] # concepts named in the prompt if len(cand) < 3: cand = CONCEPTS p = {k: 0.15 for k in cand} for t in tok: for table, w in ((CUES, 1.0), (TRAPS, 1.3)): if table.get(t) in p: p[table[t]] += w if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) # most likely choice else: e = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [v / sum(e) for v in e])] if "bicim" in tok: # the prompt contains a format declaration y = f"etiket : {k} ; kod : {CODES[k]} ; gerekce : {summary}" else: y = FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k) ot, cut = tokenize(y), 0 if len(ot) > max_out: # stop condition ot, cut, y = ot[:max_out], 1, " ".join(ot[:max_out]) return {"status": "ok", "tokens_in": len(tok), "tokens_out": len(ot), "response": y, "cut": cut} def parse_strict(y): # STRICT field parser: all three fields must be complete a = {} for part in y.split(";"): kv = part.split(":") if len(kv) == 2: a[kv[0].strip()] = kv[1].strip() complete = a.get("etiket") in TOPICS and a.get("kod") == CODES.get(a.get("etiket")) return a["etiket"] if complete and a.get("gerekce") else None def read_label(y): # LOOSE reading: the first label word in the response for s in y.lower().split(): if s in TOPICS: return s return None BASE = ("Talebi su etiketlerden birine ata : fatura , ariza , sayac , tarife , " "kesinti . Ilgili ariza kodunu ve kisa gerekcesini de bildir . Talep : ") FORMAT_DECL = "Yanitin bicimi : etiket : ; kod : ; gerekce : . " def measure(with_format, max_out=MAX_OUT, seed=SEED): CALL_COUNT[0] = 0 d = {"strict": 0, "loose": 0, "hit_rate": 0, "tokens_in": 0, "tokens_out": 0, "cut": 0} for x in TICKETS: s = call((FORMAT_DECL if with_format else "") + BASE + x["text"], " ".join(x["text"].split()[:3]), seed=seed, max_out=max_out) for a in ("tokens_in", "tokens_out", "cut"): d[a] += s[a] d["strict"] += parse_strict(s["response"]) is not None d["loose"] += read_label(s["response"]) is not None d["hit_rate"] += read_label(s["response"]) == x["topic"] for a in ("strict", "loose", "hit_rate"): d[a] /= len(TICKETS) d["calls"] = CALL_COUNT[0] return d A, B, C = measure(False), measure(True), measure(True, max_out=12) sample = call(FORMAT_DECL + BASE + TICKETS[3]["text"], " ".join(TICKETS[3]["text"].split()[:3])) print(f"FICTIONAL: {len(TICKETS)} support tickets. formatted response sample ({sample['tokens_out']} tokens)") print(f" {sample['response']}") print(f"\n{'measure':<26}{'free':>10}{'format':>14}{'format + 12':>12}") for name, k, b in (("parseable share", "strict", "{:.4f}"), ("label-read share", "loose", "{:.4f}"), ("label hit rate", "hit_rate", "{:.4f}"), ("input tokens", "tokens_in", "{:d}"), ("output tokens", "tokens_out", "{:d}"), ("cut responses", "cut", "{:d}")): print(f"{name:<26}{b.format(A[k]):>10}{b.format(B[k]):>14}{b.format(C[k]):>12}") ``` ``` FICTIONAL: 120 support tickets. formatted response sample (15 tokens) etiket : ariza ; kod : ar-14 ; gerekce : ariza kaydi actim measure free format format + 12 parseable share 0.0000 0.9750 0.0000 label-read share 0.4667 0.9750 0.4833 label hit rate 0.3083 0.6833 0.2667 input tokens 6140 8780 8780 output tokens 1392 2026 1404 cut responses 0 0 117 ``` The first row is the real number of this lesson. In the free response the number of records the strict field parser reads is **zero**: not one of the one hundred twenty responses is a three-field record. The request was clear, the label set was written in the prompt, and the fault code and the rationale were asked for separately; even so, what came out is a sentence. The second row shows that the sentences are not unreadable: in **0.4667** of the responses the loose reading finds a label word. The gap between the two is the subject of this lesson. A free response is an output that a human reads and a program cannot. When a format declaration is added the parseable share becomes **0.9750**. The remaining share comes not from the model breaking the format but from three calls caught by the rate limit; the simulator returns an empty response on every thirty-seventh call, and no record is read from an empty response. At the same time the label hit rate rises **from 0.3083 to 0.6833**, but the source of this rise is not the model deciding better. More than half of the free responses carried no label, and every record whose label could not be read was counted wrong. The format declaration does not correct the decision; it **makes the decision visible**. The cost moves in two directions at once. The input tokens rise **from 6140 to 8780**: the format declaration is twenty-two tokens and is added to the front of each of the one hundred twenty requests. The output tokens also XX, because the three-field record is longer than the free sentence. The lesson here is exactly this: the format constraint is a decision that **enlarges both token items** in this task. The third column shows the cost of taking this growth back. When the output length is limited to twelve tokens the output tokens fall to 1404, but the parseable share becomes **0.0000**: the number of cut responses is 117, and a cut record is a missing record. #### The Length Limit Is a Threshold, Not a Slope The second block sweeps the limit over a wide range and shows how the number of cut responses and the parseable share are tied together. ```python print("max out cut parseable label hit rate output tokens") for n in (24, 20, 18, 16, 14, 12, 10): d = measure(True, max_out=n) print(f"{n:>7} {d['cut']:>7} {d['strict']:>16.4f} {d['hit_rate']:>14.4f}" f" {d['tokens_out']:>15}") readable = {} for x in TICKETS: s = call(FORMAT_DECL + BASE + x["text"], " ".join(x["text"].split()[:3]), max_out=12) e = read_label(s["response"]) readable[e] = readable.get(e, 0) + 1 print(f"\nmax out 12 , readable labels in cut responses: {sorted(k for k in readable if k)}") print(f"labels over five characters: {[k for k in TOPICS if len(k) > 5]}") B2, A2 = measure(True, seed=SEED + 1), measure(False, seed=SEED + 1) print(f"\nformat constraint gain : parseable {B['strict'] - A['strict']:+.4f} ," f" hit rate {B['hit_rate'] - A['hit_rate']:+.4f}") print(f"two-run difference : parseable {B2['strict'] - B['strict']:+.4f} ," f" hit rate {B2['hit_rate'] - B['hit_rate']:+.4f}") print(f"free prompt two runs : hit rate {A['hit_rate']:.4f} / {A2['hit_rate']:.4f}") ``` ``` max out cut parseable label hit rate output tokens 24 0 0.9750 0.6833 2026 20 0 0.9750 0.6833 2026 18 30 0.7250 0.4417 1992 16 78 0.3250 0.3500 1865 14 117 0.0000 0.2917 1638 12 117 0.0000 0.2667 1404 10 117 0.0000 0.2667 1170 max out 12 , readable labels in cut responses: ['ariza', 'sayac'] labels over five characters: ['fatura', 'tarife', 'kesinti'] format constraint gain : parseable +0.9750 , hit rate +0.3750 two-run difference : parseable +0.0000 , hit rate +0.1167 free prompt two runs : hit rate 0.3083 / 0.5000 ``` The second and the third column of the chart are locked to each other. With the limit at twenty there is no cut response and the share is 0.9750; at eighteen thirty responses are cut and the share falls to 0.7250; at sixteen seventy-eight responses are cut and the share becomes 0.3250. In every row **the number of parseable responses is exactly equal to the number of responses that were neither cut nor returned as an error**. For a single record the length limit is a threshold: either the record is complete or it does not exist at all; there is no in between. At the dataset level the curve looks soft because response lengths change from record to record; the slope of the curve is a property not of the model but of **the length distribution of the rationale field**. The token column explains the appeal of this decision. Lowering the limit from twenty-four to ten brings the output tokens down **from 2026 to 1170**, that is, roughly by half. The saving is real and it follows from the decision; in return the number of records left in hand is zero. The length limit is a cost setting and **its right place is just above the full length of the record**; every token of saving below that deletes the whole record. Which label can be read in a cut response shows a tokenizer detail. With the limit at twelve the loose reading finds only the `fault` and `meter` labels; `billing`, `tariff` and `outage` never appear. The reason is the tokenizer: words longer than five characters are split into pieces, and when a cut output is joined back together these pieces remain separate words. The same length limit **behaves differently depending on the number of characters of the label**: the limit is a token threshold but its effect depends on the spelling of the word. The last three lines apply the rule of the course. The difference that the format declaration adds to parseability is **+0.9750**, while the difference between two runs of the same prompt is **+0.0000**. The measure is independent of the run and the gain is beyond dispute. The hit rate side is not as comfortable: the gain is **+0.3750**, the two-run difference **+0.1167**. Since the gain is more than three times the difference it counts as measured, but while there is a movement of 0.1167 between two runs of the same prompt, a claim of a 0.05 improvement is an unmeasured guess in this scaffolding. **The prompt is a design object; a prompt changed without measuring is not an improvement but a guess.** #### Summary - A clear request does not determine the form of the response. In the free response the record share that the strict field parser reads is **0.0000** and the share in which the loose reading finds a label is **0.4667**: the response is readable for a human and unreadable for a program. - The format declaration carries the parseable share to **0.9750** and the label hit rate **from 0.3083 to 0.6833**. The remaining 0.0250 belongs not to the model but to the three calls caught by the rate limit. - The cost is two-directional: the input tokens rise **from 6140 to 8780** and the output tokens **from 1392 to 2026**. The format constraint enlarges both token items in this task. - When the maximum output length is lowered from 24 to 10 the output tokens fall **from 2026 to 1170** and the parseable share becomes **0.0000**; in every row the number of parseable responses equals the number of responses that were neither cut nor returned as an error. - The **+0.9750** that the format declaration adds to parseability has a two-run difference of **+0.0000**; the **+0.3750** it adds to the hit rate has a two-run difference of **+0.1167**. The second measure is noisier and sets the smallest hit rate difference this scaffolding can measure. #### Next Step In this lesson what was written into the prompt was a **recipe**: the names, the order and the separator of the fields. The recipe determined the container of the response, and determining the container raised the hit rate indirectly. But the model's decision about **which ticket goes to which label** was never supported; there is not even a single example of it in the prompt. The next lesson puts **examples** in place of the recipe: a few solved tickets are written into the prompt and how the hit rate changes is counted. The number of examples is raised from zero to eight, the hit rate and the tokens are printed side by side at every step, and **the point where the gain saturates** is written. The same lesson also shows something disturbing: when the same examples are written in a different order the number changes, that is, the **ordering** of the examples is a design decision as much as the example set. ### Using Examples Source: https://academia.sh/en/courses/prompt-engineering/using-examples Zero-shot, one-shot and few-shot prompting are measured side by side on the same task. As the number of examples rises from zero to eight, the hit rate on 112 fictional support tickets goes from 0.8571 to 0.9732, but the gain saturates at the third example: from zero to three examples +0.1071 hit rate comes with 4928 extra input tokens, from three to eight examples +0.0089 hit rate comes with 8736 extra tokens, and the mean between four and eight examples, 0.9250, stays below the 0.9643 of three examples. When the same four examples are written with only their order changed, the hit rate wanders between 0.7679 and 0.9196 and the input tokens stay the same in every order. The previous lesson wrote the format of the response into the prompt and carried the machine-parseable response share from 0.0000 to 0.9750. What went into the prompt was a **recipe**: the names, the order and the separator of the fields. The recipe determined the container of the response, but it did not touch the decision itself. Which ticket goes to which label is still something only the model knows, and the prompt contains not a single example of it. This lesson puts **exemplars** in place of the recipe. Solved tickets are written into the prompt, and each of them tells the model "this text goes to this label". Three things will be measured: the hit rate gain per exemplar, the token cost per exemplar, and **where the gain stops**. At the end of the lesson the measure turns on itself once more: when the same exemplars are written with only their order changed, does the number move? - **PD48.** The model is a **simulator**, its seed is **20260218**, and no real endpoint is called. The support tickets and the fault codes are **fictional**. - **PD49.** The **format constraint** of the previous lesson is **held constant**. In every run the response is requested as a three-field record and read with the same strict field parser; so the only thing that changes in this lesson is the exemplars. - **PD50.** The measurement is made with the **most likely choice**. With sampling off, the label decision is independent of the run, and the difference that an exemplar adds does not get mixed into the sampling noise; the runs with sampling on are printed separately. - **PD51.** The exemplar pool is **separate** from the measurement set: eight solved tickets are taken into the pool and the remaining 112 tickets are measured. The first five exemplars of the pool cover the five labels once each, and the remaining three bring the second templates. - **PD52.** The simulator uses an exemplar like a **matching table** written into the prompt: every token that the exemplar text shares with the ticket raises the score of the exemplar's label. **The last exemplar weighs more**; the order effect comes from here. - **PD53.** The cost is written in **input tokens**. The exemplars are sent again with every request; the input tokens per request grow in direct proportion to the number of exemplars. #### Gain and Cost per Example ```python # SIMULATOR -- FICTIONAL. This code does not call a real model endpoint; it is a # language model simulator written with the standard library. The tickets are fictional too. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUT = 240, 24 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer: pieces of at most five characters b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def prompt_trace(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] CODES = {"fatura": "ft-21", "ariza": "ar-14", "sayac": "sy-32", "tarife": "tr-05", "kesinti": "ks-40", "odeme": "od-11", "sikayet": "sk-09", "bilgi": "bg-02", "genel": "gn-00"} TEMPLATES = {"fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"]} EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim", "acil bilgi almam gerekiyor", "kayit numaram elimde yok"] TICKETS = [] for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = TEMPLATES[k][int(r() * 3)] if r() < 0.34: m += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": m, "topic": k}) CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "borc": "odeme", "odedi": "odeme", "ucret": "odeme", "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet", "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"} TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} # misleading cues CALL_COUNT = [0] def parse(prompt): # splits the prompt into an example list and the ticket text head, _, ticket = prompt.partition("Talep : ") examples = [] for piece in head.split("Ornek : ")[1:]: m, _, e = piece.partition(" => ") examples.append((m.strip(), e.split(" .")[0].strip())) return examples, ticket def call(prompt, summary, seed=SEED, temperature=0.0, max_out=MAX_OUT): CALL_COUNT[0] += 1 tok = tokenize(prompt) empty = {"tokens_in": len(tok), "tokens_out": 0, "response": ""} if len(tok) > WINDOW: return dict(empty, status="context_overflow") if CALL_COUNT[0] % 37 == 0: return dict(empty, status="rate_limit") r = rng(seed ^ prompt_trace(prompt)) if len(tok) > 90 and r() < 0.04: # long prompt , timeout return dict(empty, status="timeout") examples, ticket = parse(prompt) tt = tokenize(ticket) cand = [k for k in CONCEPTS if k[:5] in tok] if len(cand) < 3: cand = CONCEPTS p = {k: 0.15 for k in cand} for t in tt: # ticket tokens: cue and trap for table, w in ((CUES, 1.0), (TRAPS, 1.3)): if table.get(t) in p: p[table[t]] += w for j, (m, e) in enumerate(examples): # in-context learning if e in p: w = 0.5 + (j + 1) / len(examples) # the last example weighs more for t in set(tokenize(m)): if t in tt: p[e] += 0.30 * w if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) # most likely choice else: e = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [v / sum(e) for v in e])] y = f"etiket : {k} ; kod : {CODES[k]} ; gerekce : {summary}" ot = tokenize(y) if len(ot) > max_out: ot, y = ot[:max_out], " ".join(ot[:max_out]) return {"status": "ok", "tokens_in": len(tok), "tokens_out": len(ot), "response": y} def parse_strict(y): # the strict field parser of the previous lesson a = {} for part in y.split(";"): kv = part.split(":") if len(kv) == 2: a[kv[0].strip()] = kv[1].strip() complete = a.get("etiket") in TOPICS and a.get("kod") == CODES.get(a.get("etiket")) return a["etiket"] if complete and a.get("gerekce") else None INSTRUCTION = ("Talebi su etiketlerden birine ata : fatura , ariza , sayac , tarife , " "kesinti . Yanitin bicimi : etiket : ; kod : ; " "gerekce : . ") PICKS = [1, 3, 4, 5, 0, 18, 13, 40] # first one example per label, then the second template POOL = [TICKETS[i] for i in PICKS] DATASET = [t for i, t in enumerate(TICKETS) if i not in PICKS] def build_prompt(examples, text): o = "".join(f"Ornek : {x['text']} => {x['topic']} . " for x in examples) return INSTRUCTION + o + "Talep : " + text def measure(examples, seed=SEED, temperature=0.0): CALL_COUNT[0] = 0 tin = tout = hit = bad = 0 for x in DATASET: s = call(build_prompt(examples, x["text"]), " ".join(x["text"].split()[:3]), seed=seed, temperature=temperature) tin, tout = tin + s["tokens_in"], tout + s["tokens_out"] bad += s["status"] != "ok" hit += parse_strict(s["response"]) == x["topic"] return {"hit_rate": hit / len(DATASET), "tokens_in": tin, "tokens_out": tout, "failed": bad} print(f"FICTIONAL: example pool {len(POOL)} solved tickets , measurement set {len(DATASET)} tickets") print(f"example format: Ornek : {POOL[0]['text']} => {POOL[0]['topic']} .") print(f"\n{'examples':>8} {'label':>7} {'hit rate':>8} {'input tokens':>12}" f" {'per request':>11} {'gain':>7}") previous = None for n in range(9): d = measure(POOL[:n]) gain = "-" if previous is None else f"{d['hit_rate'] - previous:+.4f}" name = POOL[n - 1]["topic"] if n else "-" print(f"{n:>8} {name:>7} {d['hit_rate']:>8.4f} {d['tokens_in']:>12}" f" {d['tokens_in'] / len(DATASET):>11.1f} {gain:>7}") previous = d["hit_rate"] ``` ``` FICTIONAL: example pool 8 solved tickets , measurement set 112 tickets example format: Ornek : fatura ayrintisini nereden gorebilirim => fatura . examples label hit rate input tokens per request gain 0 - 0.8571 6646 59.3 - 1 fatura 0.8571 8438 75.3 +0.0000 2 ariza 0.9018 9782 87.3 +0.0446 3 sayac 0.9643 11574 103.3 +0.0625 4 tarife 0.8929 13366 119.3 -0.0714 5 kesinti 0.9196 15270 136.3 +0.0268 6 fatura 0.9286 17398 155.3 +0.0089 7 ariza 0.9107 18854 168.3 -0.0179 8 sayac 0.9732 20310 181.3 +0.0625 ``` The first row is **zero-shot prompting**: the prompt contains only the instruction and the format recipe, and the hit rate is **0.8571**. The second row is **one-shot prompting** and its gain is **zero**. A single exemplar does nothing but raise the score of its own label; it says nothing about the other four labels and classifies the already correct tickets a little more correctly. Its cost is real: the input tokens per request rise **from 59.3 to 75.3**. The gain starts with **few-shot prompting**. The second exemplar adds +0.0446, the third exemplar adds +0.0625, and the hit rate becomes **0.9643**. Up to here adding exemplars pays for itself: from zero to three, **+0.1071** hit rate for **+4928** input tokens. After the fourth exemplar the chart changes direction. The fourth exemplar produces the largest single-step loss with **−0.0714**, the fifth and the sixth bring small gains, the seventh drops again, and the eighth brings it to the highest number with +0.0625. The mean of the five measurements between four and eight is **0.9250**, that is, **below the 0.9643** of three exemplars. The saturation point in this chart is **the third exemplar**. What going from three to eight adds to the hit rate is **+0.0089**, and what it adds to the input tokens is **+8736**. The first three exemplars brought roughly twenty times more hit rate per token. The reason lies in what the exemplars teach: the first three examples introduce the tokens of three separate labels and correct the tickets that were wrongly matched before. The later examples either repeat a label that is already covered or teach the **off-topic words** in their own sentences; words such as "where" and "need" are unrelated to the label and push the score in the wrong direction. Past a certain point adding examples adds not information but **noise**, and the price of noise is also paid in tokens. #### Same Examples, Different Order ```python ORDERS = [(0, 1, 2, 3), (3, 2, 1, 0), (1, 3, 0, 2), (2, 0, 3, 1), (0, 2, 1, 3), (3, 0, 2, 1)] D = POOL[:4] print("same four examples , only the order changes") v = [] for s in ORDERS: d = measure([D[i] for i in s]) v.append(d["hit_rate"]) print(f" {' > '.join(D[i]['topic'] for i in s):<34} hit rate {d['hit_rate']:.4f}" f" input {d['tokens_in']}") print(f" order range {max(v) - min(v):.4f} , input tokens equal in every order") print("\nvariability of the measure itself") for n in (0, 3, 8): a, b = measure(POOL[:n]), measure(POOL[:n], seed=SEED + 1) print(f" n={n} most likely choice : {a['hit_rate']:.4f} / {b['hit_rate']:.4f}" f" difference {b['hit_rate'] - a['hit_rate']:+.4f} , failed calls {a['failed']}") for n in (0, 3): a = measure(POOL[:n], temperature=0.6) b = measure(POOL[:n], seed=SEED + 1, temperature=0.6) print(f" n={n} sampling on : {a['hit_rate']:.4f} / {b['hit_rate']:.4f}" f" difference {b['hit_rate'] - a['hit_rate']:+.4f}") s0, s3, s8 = measure(POOL[:0]), measure(POOL[:3]), measure(POOL[:8]) mid = sum(measure(POOL[:n])["hit_rate"] for n in range(4, 9)) / 5 print(f"\nzero -> three examples : hit rate {s3['hit_rate'] - s0['hit_rate']:+.4f} ," f" input tokens {s3['tokens_in'] - s0['tokens_in']:+d}") print(f"three -> eight examples : hit rate {s8['hit_rate'] - s3['hit_rate']:+.4f} ," f" input tokens {s8['tokens_in'] - s3['tokens_in']:+d}") print(f"mean hit rate between four and eight {mid:.4f} , at three examples {s3['hit_rate']:.4f}") ``` ``` same four examples , only the order changes fatura > ariza > sayac > tarife hit rate 0.8929 input 13366 tarife > sayac > ariza > fatura hit rate 0.9196 input 13366 ariza > tarife > fatura > sayac hit rate 0.8929 input 13366 sayac > fatura > tarife > ariza hit rate 0.9196 input 13366 fatura > sayac > ariza > tarife hit rate 0.9196 input 13366 tarife > fatura > sayac > ariza hit rate 0.7679 input 13366 order range 0.1518 , input tokens equal in every order variability of the measure itself n=0 most likely choice : 0.8571 / 0.8571 difference +0.0000 , failed calls 3 n=3 most likely choice : 0.9643 / 0.9554 difference -0.0089 , failed calls 3 n=8 most likely choice : 0.9732 / 0.9464 difference -0.0268 , failed calls 3 n=0 sampling on : 0.7321 / 0.8393 difference +0.1071 n=3 sampling on : 0.8571 / 0.8750 difference +0.0179 zero -> three examples : hit rate +0.1071 , input tokens +4928 three -> eight examples : hit rate +0.0089 , input tokens +8736 mean hit rate between four and eight 0.9250 , at three examples 0.9643 ``` All six lines above contain **the same four examples**. The label set is the same, the instruction is the same, the format recipe is the same, the measurement set is the same, and **the input tokens are 13366 in all six orders**. The only thing that changes is the order in which the examples are written, and the hit rate wanders between **0.7679 and 0.9196**. The range is **0.1518**, that is, **larger than the 0.1071** that going from zero to three examples adds. The same example set, with a cost identical to the last token, can produce the difference between a good prompt and a bad prompt. The reason is that the simulator gives more weight to the last example. In the worst ordering the last example carries the `fault` label, and the `but` token in its sentence appears in all eight `billing` tickets of the measurement set; once weighted, it pulls those tickets to the `fault` side too. When the same example stands at the beginning of the list it does not have this effect. Order is a design decision and a **free** lever on the prompt: changing it brings no token cost. The sub-block gives the variability of the measure itself. With the most likely choice, the difference between two runs is at most **0.0268**; the number of failed calls is three in all three runs and comes from the rate limit. At this scale the **+0.1071** gain of going from zero to three examples is four times the two-run difference and counts as **measured**; the **+0.0089** gain of going from three to eight stays below the two-run difference and counts as **unmeasured**. When sampling is turned on the table turns around: at zero examples the difference between two runs is **0.1071**, that is, exactly as large as the gain we want to measure. Someone who tries to measure the same number in a sampled run in a single shot sees the gain and the noise at the same size and cannot tell them apart. **A prompt changed without measuring is not an improvement but a guess**, and here measuring has a precondition: the variability of the measure itself must be smaller than the difference being sought. #### Summary - Zero-shot prompting gives **0.8571**, one-shot prompting again gives **0.8571**: a single exemplar raises the input tokens per request from 59.3 to 75.3 without adding anything to the hit rate. - The gain accumulates in few-shot prompting and **saturates at the third exemplar**: from zero to three **+0.1071** hit rate and **+4928** input tokens, from three to eight **+0.0089** hit rate and **+8736** input tokens. - The mean of the five measurements between four and eight exemplars is **0.9250**, **below the 0.9643** of three exemplars; the exemplars added after the saturation point carry noise, not information. - When the same four exemplars are written with only their order changed, the hit rate wanders between **0.7679 and 0.9196**. The range is **0.1518** and the input tokens are **13366** in all six orders: order is a free design lever. - With the most likely choice the two-run difference is at most **0.0268**; with sampling on it is **0.1071** at zero exemplars. If the gain being sought is smaller than the variability of the measure itself, the measurement counts as not made. #### Next Step In these two lessons everything written into the prompt was **positive**: do this, write in this format, look at these examples. In practice, a large part of prompts contains the opposite as well: sentences that say something must not be done. Do not use a label, do not give a date, do not speak with certainty, do not make things up. Intuition says these sentences are harmless: at worst they do not work. The next lesson takes the number of this intuition and finds the opposite. The same rule is written in two separate forms, a **negative constraint** and a positive statement, and the **number of violations** each one produces is printed side by side. Naming in the prompt the very thing the prohibition forbids has a cost of its own, and this cost shows up in the number. ### Stating Constraints and Prohibitions Source: https://academia.sh/en/courses/prompt-engineering/stating-constraints-and-prohibitions The same rule is written in two separate forms and measured on 120 fictional support tickets. With no statement the violation rate is 0.2583; when a negative constraint is added it rises to 0.5333, and the number of responses carrying one of the forbidden words goes from 5 to 35. The prompt that states the same rule positively brings the violation down to 0.0167 and raises the label hit rate from 0.6667 to 0.8333. When both are written together the violation is 0.1583: adding the prohibition to the positive statement makes the number about ten times worse. The cost is written side by side; the positive statement costs 115.9 input tokens per prevented violation. In the previous two lessons everything written into the prompt was positive: do this, write in this format, look at these examples. A large part of the prompts written in practice contains the opposite as well. Do not use a label, do not give a date, do not speak with certainty, do not make things up. These sentences are called **negative constraints** (prohibitions stated in the prompt), and intuition says that at worst they will have no effect: if the model complies, good; if it does not, nothing changes. This lesson takes the number of that intuition. The same rule is written in two separate forms, once as what will not be done and once as what will be done, and the **number of violations** each produces is put side by side. The result is the first example of the second claim of this course: **a prompt addition can raise the number instead of lowering it**, and there is no way of knowing this without measuring. - **PD54.** The model is a **simulator**, its seed is **20260218**, and no real endpoint is called. The support tickets, the response templates and the rule set are **fictional**. - **PD55.** The task is **writing a short response**: the model assigns the ticket to a label and produces a one-sentence response. The response is built from one of six fictional templates. - **PD56.** The rule set is three words: `hemen` (immediately), `yarin` (tomorrow) and `genel` (general) must not appear in the response. They stand for a fictional style guide: do not make promises, do not give dates, do not put vague labels. - **PD57.** The **violation measure** is explicit: if one of these three words appears in the response, the record is in violation. How many responses each word appears in is also counted separately. - **PD58.** The simulator chooses the response template according to the **tokens that appear in the prompt**. Every template has its own distinguishing tokens, and when these tokens appear in the prompt the score of that template rises. The model does not distinguish whether a token is mentioned **positively or negatively**. - **PD59.** Four prompts are compared: no statement, with a negative constraint, with a positive statement, and both together. The task definition, the label set and the ticket set are the same in all four. - **PD60.** The cost is written in **input tokens**, and the unit cost **per prevented violation** is computed as well. #### Two Writings of the Same Rule ```python # SIMULATOR -- FICTIONAL. This code does not call a real model endpoint; it is a # language model simulator written with the standard library. The tickets and the response templates are fictional too. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUT = 240, 24 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer: pieces of at most five characters b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def prompt_trace(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] TEMPLATES = {"fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"]} EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim", "acil bilgi almam gerekiyor", "kayit numaram elimde yok"] TICKETS = [] for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = TEMPLATES[k][int(r() * 3)] if r() < 0.34: m += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": m, "topic": k}) CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "borc": "odeme", "odedi": "odeme", "ucret": "odeme", "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet", "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"} TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} CALL_COUNT = [0] # FICTIONAL rule set: three words that must not appear in the response. FORBIDDEN = ["hemen", "yarin", "genel"] # Every response template comes with a base weight and its own distinguishing tokens. RESPONSES = [("kaydiniz {k} konusunda acildi , inceleme suruyor", 1.2, ["incel", "acild"]), ("{k} talebiniz ilgili ekibe yonlendirildi", 1.1, ["ilgil", "ekibe"]), ("bu {k} kaydi hemen ele alinacak", 0.5, ["hemen"]), ("{k} kaydiniz yarin sonuclandirilacak", 0.4, ["yarin"]), ("talebiniz genel bir basvuru olarak alindi", 0.3, ["genel", "basvu"]), ("{k} kaydiniz durumu inceleniyor", 1.0, ["durum", "incel"])] def call(prompt, seed=SEED, temperature=0.6, max_out=MAX_OUT): CALL_COUNT[0] += 1 tok = tokenize(prompt) empty = {"tokens_in": len(tok), "tokens_out": 0, "response": "", "template": -1} if len(tok) > WINDOW: return dict(empty, status="context_overflow") if CALL_COUNT[0] % 37 == 0: return dict(empty, status="rate_limit") r = rng(seed ^ prompt_trace(prompt)) cand = [k for k in CONCEPTS if k[:5] in tok] if len(cand) < 3: cand = CONCEPTS p = {k: 0.15 for k in cand} for t in tok: for table, w in ((CUES, 1.0), (TRAPS, 1.3)): if table.get(t) in p: p[table[t]] += w e = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [v / sum(e) for v in e])] # Template score: base weight + the number of times the template's tokens appear in the prompt. score = [t + 0.85 * sum(a in tok for a in marks) for _, t, marks in RESPONSES] q = [math.exp(x / temperature) for x in score] i = pick(r(), [x / sum(q) for x in q]) y = RESPONSES[i][0].format(k=k) ot = tokenize(y) if len(ot) > max_out: ot, y = ot[:max_out], " ".join(ot[:max_out]) return {"status": "ok", "tokens_in": len(tok), "tokens_out": len(ot), "response": y, "template": i} def read_label(y): for s in y.lower().split(): if s in TOPICS: return s return None TASK = ("Talebi su etiketlerden birine ata ve kisa bir yanit yaz : fatura , ariza , " "sayac , tarife , kesinti . Talep : ") PROHIBITION = "Yanitinda su sozcukleri kullanma : hemen , yarin , genel . " POSITIVE = ("Yanitinda yalnizca sunlar olsun : kaydin acildigi , ilgili ekibe " "yonlendirildigi ve durumun inceleniyor olmasi . ") CASES = (("no statement", ""), ("prohibition", PROHIBITION), ("positive", POSITIVE), ("prohibition + positive", PROHIBITION + POSITIVE)) def measure(prefix, seed=SEED): CALL_COUNT[0] = 0 d = {"violation": 0, "labeled": 0, "hit_rate": 0, "tokens_in": 0, "tokens_out": 0} counts, templates = {w: 0 for w in FORBIDDEN}, [0] * len(RESPONSES) for x in TICKETS: s = call(prefix + TASK + x["text"], seed=seed) d["tokens_in"], d["tokens_out"] = d["tokens_in"] + s["tokens_in"], d["tokens_out"] + s["tokens_out"] if s["template"] >= 0: templates[s["template"]] += 1 found = [w for w in FORBIDDEN if w in s["response"].split()] for w in found: counts[w] += 1 d["violation"] += bool(found) d["labeled"] += read_label(s["response"]) is not None d["hit_rate"] += read_label(s["response"]) == x["topic"] for a in ("violation", "labeled", "hit_rate"): d[a] /= len(TICKETS) return dict(d, counts=counts, templates=templates, calls=CALL_COUNT[0]) print(f"FICTIONAL: {len(TICKETS)} support tickets , forbidden words {FORBIDDEN}") print(f"\n{'prompt':<23}{'violation':>10}{'labeled':>10}{'hit rate':>10}{'input':>8}" f"{'hemen':>7}{'yarin':>7}{'genel':>7}") O = {} for name, prefix in CASES: d = O[name] = measure(prefix) print(f"{name:<23}{d['violation']:>10.4f}{d['labeled']:>10.4f}{d['hit_rate']:>10.4f}" f"{d['tokens_in']:>8}" + "".join(f"{d['counts'][w]:>7}" for w in FORBIDDEN)) ``` ``` FICTIONAL: 120 support tickets , forbidden words ['hemen', 'yarin', 'genel'] prompt violation labeled hit rate input hemen yarin genel no statement 0.2583 0.9000 0.6667 5060 5 17 9 prohibition 0.5333 0.8583 0.5750 6740 35 16 13 positive 0.0167 0.9667 0.8333 8420 1 0 1 prohibition + positive 0.1583 0.9417 0.6500 10100 12 3 4 ``` The first row is the baseline: with no statement, a share of **0.2583** of the responses carries one of the forbidden words. The second row is the number of this lesson. When the negative constraint is added the violation rate becomes **0.5333**, that is, **more than double**. Writing the prohibition does not prohibit the forbidden thing; it doubles it. The three columns on the right show where the increase gathers: the number of responses carrying the word `hemen` goes **from 5 to 35**, `genel` goes from 9 to 13 and `yarin` goes from 17 to 16. The increase is not evenly distributed, and the only fall is one unit. The third row is the positive writing of the same rule. The prompt never mentions which words are forbidden; instead it says **what will be in the response**. The violation rate becomes **0.0167**, one fifteenth of the baseline. The side effect is also in the positive direction: the share of responses with a readable label goes from 0.9000 to **0.9667** and the label hit rate goes from 0.6667 to **0.8333**. The same rule, the same model, the same task and the same set; the only thing that changes is the sentence of the rule. The fourth row is the writing most often seen in practice: first what is wanted is said, then "and do not do these" is added. The violation rate becomes **0.1583**, about **ten times** the 0.0167 that the positive statement gives alone. When the prohibition sentence is added next to a positive prompt it carries its own damage as well. The label hit rate also falls back from 0.8333 to 0.6500. #### Why the Prohibition Works in Reverse ```python print("template none prohibit positive both") for i, (text, _, marks) in enumerate(RESPONSES): label = ("forbidden: " + [w for w in FORBIDDEN if w in text][0]) if any( w in text for w in FORBIDDEN) else "allowed: " + marks[0] print(f"{label:<30}" + "".join(f"{O[name]['templates'][i]:>9}" for name, _ in CASES)) print("\nrobustness of the increase across runs") for sd in (SEED, SEED + 1, SEED + 2): v = [measure(prefix, seed=sd)["violation"] for _, prefix in CASES] print(f" seed {sd}: " + " ".join(f"{x:.4f}" for x in v)) v0 = [measure(prefix)["violation"] for _, prefix in CASES] v1 = [measure(prefix, seed=SEED + 1)["violation"] for _, prefix in CASES] print(" two-run difference: " + " ".join(f"{abs(a - b):.4f}" for a, b in zip(v0, v1))) n = len(TICKETS) base = O["no statement"] print("\ncost") for name, _ in CASES[1:]: d, diff = O[name], round((O[name]["violation"] - base["violation"]) * n) extra = d["tokens_in"] - base["tokens_in"] unit = f"{extra / abs(diff):.1f}" if diff else "-" print(f" {name:<22} violations {round(base['violation'] * n):>3} -> {round(d['violation'] * n):>3}" f" ({diff:+d}) , input tokens {extra:+d} , per violation {unit}") ``` ``` template none prohibit positive both allowed: incel 39 18 19 52 allowed: ilgil 26 16 77 23 forbidden: hemen 5 35 1 12 forbidden: yarin 17 16 0 3 forbidden: genel 9 13 1 4 allowed: durum 21 19 19 23 robustness of the increase across runs seed 20260218: 0.2583 0.5333 0.0167 0.1583 seed 20260219: 0.2750 0.4167 0.0000 0.1167 seed 20260220: 0.3833 0.4917 0.0083 0.1167 two-run difference: 0.0167 0.1167 0.0167 0.0417 cost prohibition violations 31 -> 64 (+33) , input tokens +1680 , per violation 50.9 positive violations 31 -> 2 (-29) , input tokens +3360 , per violation 115.9 prohibition + positive violations 31 -> 19 (-12) , input tokens +5040 , per violation 420.0 ``` The chart above makes the mechanism clear. In the prompt with no statement the `hemen` template is chosen 5 times; in the prompt with the negative constraint **35 times**. The prohibition sentence wrote the token of this template's text into the prompt, and the simulator processed that token as **a word it saw in the prompt**. The model does not distinguish whether a word is mentioned together with "do not use" or together with "use"; what it sees is the **presence** of the word. The positive statement uses the same mechanism in the opposite direction: the allowed template with the `ilgil` token goes from 26 **to 77**, because this time the words that appear in the prompt are the words of the allowed template. The middle block tests whether the difference is independent of the run. The ordering is the same in all three seeds: the negative constraint is above the baseline, and the positive statement is far below the baseline. The difference between two runs is **0.0167** in the prompt with no statement, **0.0167** in the positive statement and **0.1167** in the negative constraint. The **+0.2750** that the negative constraint adds is larger than twice its own run difference and counts as measured. The **-0.2416** that the positive statement adds is fourteen times the run difference and is beyond dispute. That the run difference of the negative constraint is seven times larger than the other two is a finding in itself: the prohibition does not only worsen the result, it makes it **more unstable** too. The sub-block gives the cost. The negative constraint adds 1680 input tokens and raises the number of violations from 31 to 64; every token added pushes the number in the wrong direction. The positive statement adds 3360 input tokens and brings the violations from 31 down to 2: **115.9 input tokens per prevented violation**. Adding the prohibition to the positive statement spends 5040 tokens and prevents only 12 violations, that is, **420.0 tokens per prevented violation**, when there is a prompt that does the same job at 115.9 tokens per prevented violation. The prompt that spends twice as many tokens is the right prompt here, and it is possible to say this only by running both and counting. **A prompt changed without measuring is not an improvement but a guess**; in this lesson the guess came out in the opposite direction of the number. #### Summary - The same rule was measured with two separate writings. The violation rate is **0.2583** in the prompt with no statement and **0.5333** in the prompt with the negative constraint: writing the prohibition raised the violation to more than double. - The increase gathers in the forbidden words: the number of responses carrying `hemen` goes **from 5 to 35**, `genel` goes from 9 to 13 and `yarin` goes from 17 to 16. - The positive statement builds the same rule without mentioning any forbidden word and brings the violation down to **0.0167**; the label hit rate goes from 0.6667 to **0.8333**. - When both are written together the violation becomes **0.1583**, about ten times what the positive statement gives alone. The prohibition sentence carries its own damage inside a positive prompt too. - The cost: the negative constraint adds 33 violations with **+1680** input tokens, the positive statement prevents 29 violations with **+3360** input tokens (**115.9 tokens per prevented violation**), and both together prevent only 12 violations with **+5040** tokens. #### Next Step In four lessons four separate design decisions were measured, and each was tried alone, in a single shot. A real prompt is not written this way. A prompt is written, measured, a part of it is changed, measured again, and this loop continues until the number stops improving. The next lesson puts the loop itself under measurement: in how many iterations the gain saturates and, the more disturbing question, **when the number of the set looked at during the iterations and the number of a set never looked at begin to separate**. This lesson is also the closing of the topic and ties the total of the eight lessons together with a question. ### Prompt Iteration Source: https://academia.sh/en/courses/prompt-engineering/prompt-iteration The loop of improving by measuring is itself put under measurement. Ten candidate changes are searched greedily over eight iterations on a validation set of 30 records: the validation hit rate rises from 0.4000 to 1.0000 and hits the ceiling at the fifth iteration, and the test hit rate reaches its highest value of 0.9333 at the same point and falls to 0.8111 at the eighth iteration. The mean of the two-run difference on validation is 0.1037 and the mean step gain is 0.0625: only two of the eight iterations are larger than their own measurement noise. When the search is repeated in another run, only one of the eight steps chooses the same change at the same place. The cost is 2640 calls, and three of the eight chosen changes are irrelevant to the task. In the previous four lessons four design decisions were measured, and each was tried alone, one at a time. A real prompt is not written this way. A prompt is written, measured, changed in one place, measured again; this loop goes on until the number stops improving. The name of the loop is **prompt iteration**, and this lesson puts the loop itself under measurement. To put it under measurement is to carry here a separation established in the Introduction to Machine Learning course of the previous curriculum: if you decide by **looking at** a number, that number is now the number of the set you decide on and does not represent general performance. There the separation was built with the training, validation and test sets. Here no model is trained, a **prompt is searched**; but the separation is the same separation. The set looked at in every iteration is the **validation** set, and the set never looked at is the **test** set. - **PD61.** The model is a **simulator**, its seed is **20260218**, and no real endpoint is called. The support tickets are **fictional**. - **PD62.** The 120 fictional tickets are split in two: **30 records validation**, **90 records test**. The search sees only the validation set; the test set enters no decision. - **PD63.** There are ten **candidate changes**. Five are relevant to the task (label set, format constraint, two examples, two more examples, positive statement), five are irrelevant (polite opening, write short, thanks, note line, reader note). The irrelevant ones change **only the text** of the prompt. - **PD64.** The search is **greedy**: at every iteration all the remaining candidates are tried on validation, the one that gives the best number is chosen and added to the prompt permanently. - **PD65.** The cost is written in **number of calls**. Every candidate tried spends as many calls as the size of the validation set, and these calls cannot be taken back. #### The Two Numbers of Eight Iterations ```python # SIMULATOR -- FICTIONAL. This code does not call a real model endpoint; it is a # language model simulator written with the standard library. The tickets are fictional too. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUT = 240, 24 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer: pieces of at most five characters b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def prompt_trace(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] TEMPLATES = {"fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"]} EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim", "acil bilgi almam gerekiyor", "kayit numaram elimde yok"] TICKETS = [] for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = TEMPLATES[k][int(r() * 3)] if r() < 0.34: m += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": m, "topic": k}) CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "borc": "odeme", "odedi": "odeme", "ucret": "odeme", "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet", "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"} TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} FREE_FORM = ["talebiniz {k} konusuyla ilgili gorunuyor", "bu kayit {k} basligina girer , ilgili ekibe yonlendirildi", "kaydiniz alindi , konuyu inceleyip donus yapacagiz"] CALL_COUNT = [0] def parse(prompt): head, _, ticket = prompt.partition("Talep : ") o = [] for part in head.split("Ornek : ")[1:]: m, _, e = part.partition(" => ") o.append((m.strip(), e.split(" .")[0].strip())) return o, ticket def call(prompt, seed=SEED, temperature=0.6, max_out=MAX_OUT): CALL_COUNT[0] += 1 tok = tokenize(prompt) empty = {"tokens_in": len(tok), "tokens_out": 0, "response": ""} if len(tok) > WINDOW: return dict(empty, status="context_overflow") if CALL_COUNT[0] % 37 == 0: return dict(empty, status="rate_limit") r = rng(seed ^ prompt_trace(prompt)) examples, ticket = parse(prompt) tt = tokenize(ticket) cand = [k for k in CONCEPTS if k[:5] in tok] if len(cand) < 3: cand = CONCEPTS p = {k: 0.15 for k in cand} for t in tt: for table, w in ((CUES, 1.0), (TRAPS, 1.3)): if table.get(t) in p: p[table[t]] += w for j, (m, e) in enumerate(examples): if e in p: w = 0.5 + (j + 1) / len(examples) for t in set(tokenize(m)): if t in tt: p[e] += 0.30 * w e = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [v / sum(e) for v in e])] y = (f"etiket : {k}" if "bicim" in tok else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k)) ot = tokenize(y) if len(ot) > max_out: ot, y = ot[:max_out], " ".join(ot[:max_out]) return {"status": "ok", "tokens_in": len(tok), "tokens_out": len(ot), "response": y} def read_label(y): for s in y.lower().split(): if s in TOPICS: return s return None VALIDATION, TEST = TICKETS[:30], TICKETS[30:] # 30 validation , 90 test BASE = "Su destek talebini yanitla . " CHANGES = [ ("label set", "Talebi su etiketlerden birine ata : fatura , ariza , sayac , " "tarife , kesinti . "), ("format constraint", "Yanitin bicimi : etiket : . "), ("two examples", "Ornek : fatura ayrintisini nereden gorebilirim => fatura . " "Ornek : ariza kaydi actim ama ekip gelmedi => ariza . "), ("two more examples", "Ornek : sayac okumasi yanlis girilmis endeks tutmuyor => sayac . " "Ornek : hangi tarife basamagindayim ogrenmek istiyorum => tarife . "), ("positive statement", "Yanitinda yalnizca konu etiketi olsun . "), ("polite opening", "Lutfen dikkatli davran . "), ("write short", "Yanitini kisa tut . "), ("thanks", "Tesekkurler . "), ("note line", "Not : kayit sistemine islenecek . "), ("reader note", "Yanit destek ekibi tarafindan okunacak . "), ] def measure(parts, dataset, seed=SEED): CALL_COUNT[0] = 0 d = 0 for x in dataset: s = call(BASE + "".join(parts) + "Talep : " + x["text"], seed=seed) d += read_label(s["response"]) == x["topic"] return d / len(dataset), CALL_COUNT[0] def search(seed, steps=8): """At every iteration tries all candidates on VALIDATION and chooses the best.""" chosen, remaining, calls, trace = [], list(CHANGES), 0, [] for _ in range(steps): best, name = None, None for a, part in remaining: v, c = measure(chosen + [part], VALIDATION, seed) calls += c if best is None or v > best: best, name = v, (a, part) chosen.append(name[1]) remaining = [x for x in remaining if x[0] != name[0]] dv, c1 = measure(chosen, VALIDATION, seed) sv, c2 = measure(chosen, TEST, seed) calls += c1 + c2 trace.append((name[0], dv, sv, calls)) return trace d0, c0 = measure([], VALIDATION) s0, c1 = measure([], TEST) print("iteration selected change validation test gap total calls") print(f"{0:>9} {'-':<20} {d0:>10.4f} {s0:.4f} {d0 - s0:+.4f} {c0 + c1:>11}") TRACE = search(SEED) for i, (name, dv, sv, cg) in enumerate(TRACE, 1): print(f"{i:>9} {name:<20} {dv:>10.4f} {sv:.4f} {dv - sv:+.4f} {cg + c0 + c1:>11}") ``` ``` iteration selected change validation test gap total calls 0 - 0.4000 0.3778 +0.0222 120 1 note line 0.7000 0.5889 +0.1111 540 2 format constraint 0.8000 0.5889 +0.2111 930 3 two examples 0.8333 0.8556 -0.0222 1290 4 label set 0.9667 0.8556 +0.1111 1620 5 positive statement 1.0000 0.9333 +0.0667 1920 6 write short 0.9667 0.9111 +0.0556 2190 7 two more examples 0.9667 0.8667 +0.1000 2430 8 thanks 0.9000 0.8111 +0.0889 2640 ``` The validation column is the usual view of a search record: **from 0.4000 to 1.0000**, an uninterrupted rise up to the fifth iteration. Someone who looks at this column alone would say that the search has made the prompt fully accurate. The test column does not tell the same story. The highest test number is **0.9333** and it comes at the fifth iteration; the next three iterations bring it down to **0.8111**. Every change made after the fifth iteration was chosen to protect validation and hurt on test. The gap column carries the difference between the two sets. At the start it is **+0.0222**, that is, the two sets say almost the same thing. At the second iteration it rises to **+0.2111**: validation says 0.8000 while test is still 0.5889. At this point every decision made by looking at validation is speaking about a set that has nothing to do with it. The source of the gap is the smallness of the set -- in thirty records the correction of one record moves the number by 0.0333, and the random correction of three records looks like an improvement. The first row is the clearest example of this. The first change the search chooses is the **note line**: a sentence that adds no information to the task, that says nothing of the label set, the format or the example. It brought **+0.3000** on validation and was the largest single step gain measured in the whole search. Three of the eight changes are of this kind. A greedy search chooses what raises the number; it does not ask why it raised the number. #### The Reliability of the Search Itself ```python SECOND = search(SEED + 1) print("same search , another run") for i, ((a1, *_), (a2, d2, s2, _)) in enumerate(zip(TRACE, SECOND), 1): print(f"{i:>2} {a1:<18} -> {a2:<18} validation {d2:.4f} test {s2:.4f}") print(f"changes the two searches chose in the same order: " f"{sum(a == b for (a, *_), (b, *_) in zip(TRACE, SECOND))}/8") sel, fv, ft = [], [], [] for name, *_ in [(None,)] + TRACE: if name: sel.append(dict(CHANGES)[name]) fv.append(abs(measure(sel, VALIDATION)[0] - measure(sel, VALIDATION, SEED + 1)[0])) ft.append(abs(measure(sel, TEST)[0] - measure(sel, TEST, SEED + 1)[0])) gv = sum(fv) / len(fv) print(f"\ntwo-run difference -- validation (30 records) mean {gv:.4f} , max {max(fv):.4f}") print(f"two-run difference -- test (90 records) mean {sum(ft) / len(ft):.4f} ," f" max {max(ft):.4f}") vals, tests = [d0] + [x[1] for x in TRACE], [s0] + [x[2] for x in TRACE] steps = [vals[i] - vals[i - 1] for i in range(1, 9)] print(f"iterations whose step gain exceeds the two-run difference: " f"{sum(x > gv for x in steps)}/8 , mean step gain {sum(steps) / 8:+.4f}") print(f"\nvalidation max {max(vals):.4f} (iteration {vals.index(max(vals))}) ," f" test max {max(tests):.4f} (iteration {tests.index(max(tests))})") print(f"5 -> 8 : validation {vals[5]:.4f} -> {vals[8]:.4f} , test {tests[5]:.4f} -> {tests[8]:.4f}") print(f"gap max {max(d - s for d, s in zip(vals, tests)):+.4f} , at iteration 8 " f"{vals[8] - tests[8]:+.4f}") irrelevant = ("polite opening", "write short", "thanks", "note line", "reader note") print(f"search cost {TRACE[-1][3] + c0 + c1} calls , of the eight selected changes " f"{sum(a in irrelevant for a, *_ in TRACE)} irrelevant to the task") ``` ``` same search , another run 1 note line -> write short validation 0.7000 test 0.5667 2 format constraint -> label set validation 0.6333 test 0.4444 3 two examples -> format constraint validation 0.7667 test 0.7333 4 label set -> two examples validation 0.9667 test 0.8778 5 positive statement -> polite opening validation 0.9667 test 0.9111 6 write short -> note line validation 0.8667 test 0.7111 7 two more examples -> two more examples validation 0.9000 test 0.8889 8 thanks -> reader note validation 0.9333 test 0.8333 changes the two searches chose in the same order: 1/8 two-run difference -- validation (30 records) mean 0.1037 , max 0.1667 two-run difference -- test (90 records) mean 0.0728 , max 0.1778 iterations whose step gain exceeds the two-run difference: 2/8 , mean step gain +0.0625 validation max 1.0000 (iteration 5) , test max 0.9333 (iteration 5) 5 -> 8 : validation 1.0000 -> 0.9000 , test 0.9333 -> 0.8111 gap max +0.2111 , at iteration 8 +0.0889 search cost 2640 calls , of the eight selected changes 3 irrelevant to the task ``` The chart above reruns the search **in the same scaffolding, changing only the seed**. Of the eight steps, **only one** chooses the same change at the same place. The second run chooses "write short" at the first step, the label set at the second step and reaches "polite opening" at the fifth step. The two searches produced two different prompts; both look good on their own validation set, and both are the product of the same scaffolding. **The search finds not what it looks for but the run.** The sub-section gives the number of this. On the validation set the difference between two runs of the same prompt is **0.1037** on average, 0.1667 at most. The mean gain of the search per step is **+0.0625**. In **only two** of the eight iterations the step gain is larger than its own measurement noise; the remaining six steps count as **unmeasured** by the rule of the course, and in spite of this they were added to the prompt permanently. On the test set the same difference is **0.0728** on average, that is, on a set three times larger the measure is somewhat calmer. A small validation set is both cheaper and produces by itself the noise that the search will chase. The saturation point is the **fifth iteration**: validation hits 1.0000 there and cannot rise any further, and test takes its highest value there too. The next three iterations spend **720 extra calls** and bring validation down to 0.9000 and test from 0.9333 to 0.8111. The total cost of the search is **2640 calls**, and the larger part of these calls went to candidates that were not chosen. A prompt search is subject to the same law as a tuning search: as the budget grows, the gain on validation does not carry over to test. **A prompt changed without measuring is not an improvement but a guess**; if the measured set is wrong, measuring too is guessing. #### Summary - The greedy search carried the validation hit rate **from 0.4000** to **1.0000** at the fifth iteration and pulled it back to 0.9000 at the eighth; the test hit rate in the same period rose **from 0.3778** to its highest **0.9333** and fell to **0.8111**. - The gain **saturated at the fifth iteration**. The next three iterations spent 720 extra calls and lowered test by 0.1222; the gap between the two sets took its largest value at the second iteration, **+0.2111**. - The first and largest step of the search (**+0.3000**) came from a sentence irrelevant to the task; **three** of the eight chosen changes add no information to the task. - On validation the two-run difference is **0.1037** on average and the step gain **+0.0625** on average: only **two** of the eight iterations are larger than their own measurement noise. On test the same difference is **0.0728**. - When the same search is run with another seed, **only one** of the eight steps chooses the same change at the same place; the total cost is **2640 calls**. #### Next Step In eight lessons the prompt turned into a design object. It was split into its components, it gained a permanent instruction, it was opened up at the level of role and ticket, its format and length were determined, it took examples, its rules were written positively and it was iterated by measuring. At every step the only thing that changed was the **text**, and the cost of every step was written in tokens, calls or violations. Their common side is also clear: all of them arranged **what to say** to the model. None of them asked the model **how to think**. In these eight lessons the model answered every ticket in a single move: it took the prompt, chose a label, wrote a sentence. There was no step in between. Yet a ticket such as calculating the difference between two billing periods and finding which tariff the overpayment was made under is not something to be solved in a single move; it asks for a few intermediate steps, and those steps must be produced by the model itself. The next topic takes this up and in its first lesson sets up the second claim of the course: asking for an intermediate step **multiplies the tokens** and does not pay off in every task. The same technique pays off in a multi-step calculation, while in a single-step classification it raises the cost and leaves the number where it was, and both are printed. ## Reasoning Techniques Measuring, on two tasks at once, what asking the model to take more steps gains and what it costs: chain of thought paying off on multi-step calculation but multiplying tokens without a gain on single-step classification; self-consistency accuracy and call cost as the number of samples goes from one to nine, and consensus breaking on a wrong answer; the number of calls growing as the branching factor rises and the point where the gain stops; the difference between a two-stage prompt that asks for the principle first and the solution second; the share of responses with a valid action when thought and action are requested in the same prompt; the token cost of intermediate steps that are invisible before the response and the task where the difference stays unmeasurable; the overlap between the errors of different prompts; the gain in validation not carrying over to the test as the search budget grows. ### Chain of Thought Source: https://academia.sh/en/courses/prompt-engineering/chain-of-thought Measuring, on two separate tasks, what happens when a prompt asks the model for the intermediate steps instead of the result: on the multi-step fictional tariff calculation accuracy rises from 0.3000 to 0.8833 and tokens per correct response fall from 243.2 to 150.1, while on single-step support ticket classification the difference changes sign across the three runs as -0.0500, -0.0417 and +0.0750 and stays inside the run band. Total tokens grow about 1.8 times on both tasks; in classification the cost per correct response rises from 59.1 to 114.1 tokens. The prompt design topic spent eight lessons arranging **what to say** in a prompt: instruction, role, measurable request, format, example, negative constraint and iterating on all of them by measuring. None of them asked **how the model should think**. The prompt the eighth lesson left behind still asks a single thing and expects a single response. Chain of thought fills this gap: asking the model not for the result but for the **intermediate steps** that lead to it. This lesson measures that one-sentence change on **two tasks**, because the second claim of the course is built here: reasoning techniques are expensive and **do not pay off on every task.** - **RT1.** The code does not call a real model endpoint; it calls a language model **simulator** written with the standard library. Seed **20260218**. - **RT2.** The support tickets and the tariff are **fictional**: 120 short support tickets, 60 account records. - **RT3.** The tiers of the fictional tariff are 10, 25 and 40 m³; on top of them come a fixed fee and a discount that depends on the subscriber type. The tariff description is inside the prompt and counts toward the token cost. - **RT4.** The simulator has two calculation paths: the **one-move** path that picks one of the shortcut patterns, and the **step-by-step** path that calculates each tier separately and carries a 0.05 slip margin per step. This distinction is a modeling assumption. - **RT5.** In classification, asking for step by step changes only the **format** of the response; the label scoring stays the same. The criterion is **exact equality** in the calculation and **label accuracy** in classification. - **RT6.** The stepwise response is long; the maximum output length is raised from 24 to **96 tokens**, otherwise the stop condition cuts the result off. This expansion is also part of the cost. - **RT7.** **Run band**: the same prompt is run on three seeds, and the difference between the highest and the lowest accuracy is the band. A change whose difference stays inside the band **counts as unmeasured**. #### Two Versions of the Same Prompt Three numbers are fixed: the **previous prompt**, the **changed prompt**, the **cost**. The previous prompt is the version the prompt design topic left behind: task defined, tariff given, format imposed. The changed prompt adds a single sentence to it — first write each tier separately, step by step. ```python # SIMULATOR. This code does not call a real model endpoint; it is a language model # simulator written with the standard library. The tickets and the tariff are FICTIONAL too. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUT, MAX_OUT_STEPS = 240, 24, 96 # context window and output limits def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): # FICTIONAL tokenizer: a piece has at most five characters out = [] for piece in text.lower().replace(",", " , ").split(): while len(piece) > 5: out.append(piece[:5]) piece = piece[5:] out.append(piece) return out def trace_id(text): # depends on the prompt's content, independent of the run h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] PATTERNS = { "fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"], } EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim", "acil bilgi almam gerekiyor", "kayit numaram elimde yok"] TICKETS = [] for i in range(120): r = rng(SEED + i) topic = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] text = PATTERNS[topic][int(r() * 3)] if r() < 0.34: text += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": text, "topic": topic}) CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti"} TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} # misleading cues RATIONALE = ("birinci adim : talep metnindeki isaretler tarandi . ikinci adim : en guclu " "isaret {topic} kavramini gosteriyor . ucuncu adim : etiket {topic}") # FICTIONAL tariff: tier upper bound (m3) and unit price, fixed fee, subscriber type discount TIERS, FIXED_FEE = [(10, 3), (25, 5), (40, 8), (10 ** 6, 12)], 20 SUBSCRIBER_TYPES = {"konut": 0, "tarim": 20, "resmi": 10} CASES = [] for i in range(60): r = rng(SEED + 900 + i) m3 = 6 + int(r() * 52) sub_type = sorted(SUBSCRIBER_TYPES)[pick(r(), [0.40, 0.34, 0.26])] CASES.append({"id": 7001 + i, "m3": m3, "type": sub_type, "text": f"tuketim {m3} m3 abone tipi {sub_type} tutar ne kadar"}) def tiered_total(m3): # total of the consumption split across tiers low, left, total = 0, m3, 0 for high, unit in TIERS: share = min(left, high - low) total, left, low = total + share * unit, left - share, high if left <= 0: break return total def exact_answer(m3, sub_type): # the task's correct answer, outside the simulator return ((tiered_total(m3) + FIXED_FEE) * (100 - SUBSCRIBER_TYPES[sub_type])) // 100 COUNTER = [0] def _parse_task(toks): # consumption and subscriber type from the ticket at the end return (next((int(u) for u in reversed(toks) if u.isdigit()), 0), next((t for t in reversed(toks) if t in SUBSCRIBER_TYPES), "konut")) def _step_by_step(m3, sub_type, r, slip=0.05): # each tier separate, a slip per step low, left, total, steps = 0, m3, 0, [] for i, (high, unit) in enumerate(TIERS): share = min(left, high - low) u = TIERS[i + 1][1] if r() < slip and i + 1 < len(TIERS) else unit total, left, low = total + share * u, left - share, high steps.append(f"basamak {i + 1} , {share} m3 , birim {u} , ara toplam {total}") if left <= 0: break if r() < slip: steps.append(f"sabit bedel atlandi ara toplam {total}") else: total += FIXED_FEE steps.append(f"sabit bedel {FIXED_FEE} ara toplam {total}") discount = 0 if r() < slip else SUBSCRIBER_TYPES[sub_type] total = (total * (100 - discount)) // 100 return " . ".join(steps + [f"indirim {discount} sonuc {total}"]) def _one_move(m3, sub_type, r): # shortcut patterns that skip the tiers d, disc = next(u for h, u in TIERS if m3 <= h), SUBSCRIBER_TYPES[sub_type] options = [exact_answer(m3, sub_type), ((m3 * d + FIXED_FEE) * (100 - disc)) // 100, tiered_total(m3) + FIXED_FEE, (tiered_total(m3) * (100 - disc)) // 100, m3 * TIERS[0][1] + FIXED_FEE] return options[pick(r(), [0.16, 0.30, 0.22, 0.20, 0.12])] def call(prompt, seed=SEED, temperature=0.6, max_out=MAX_OUT): COUNTER[0] += 1 toks = tokenize(prompt) if len(toks) > WINDOW: return {"status": "context_overflow", "input": len(toks), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: return {"status": "rate_limit", "input": len(toks), "output": 0, "response": ""} r = rng(seed ^ trace_id(prompt)) stepwise = "adim" in toks # does the prompt ask for a step-by-step solution if "hesap" in toks: # multi-step calculation task m3, sub_type = _parse_task(toks) y = (_step_by_step(m3, sub_type, r) if stepwise else f"sonuc {_one_move(m3, sub_type, r)}") else: # single-step classification task p = {k: 0.15 for k in TOPICS} # the label set is already written in the prompt for t in toks: for table, w in ((CUES, 1.0), (TRAPS, 1.3)): if table.get(t) in p: p[table[t]] += w e = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [v / sum(e) for v in e])] y = RATIONALE.format(topic=k) if stepwise else k out = tokenize(y) if len(out) > max_out: # stop condition out = out[:max_out] y = " ".join(out) return {"status": "ok", "input": len(toks), "output": len(out), "response": y} def read_label(y): # the first label in the response return next((s for s in y.split() if s in TOPICS), None) def read_number(y): # the last number in the response return next((int(x) for x in reversed(y.split()) if x.isdigit()), None) TARIFF = ("Tarife: ilk 10 m3 birim 3 , sonraki 15 m3 birim 5 , sonraki 15 m3 birim 8 , " "40 ustu birim 12 . Sabit bedel 20 . Abone tipi indirimi yuzde konut 0 , " "tarim 20 , resmi 10 . Tutari hesapla . ") CALC_BEFORE = TARIFF + "Son satira sadece sonucu yaz . Talep: " CALC_STEP = TARIFF + "Once her basamagi ayri ayri adim adim yaz , sonra sonucu yaz . Talep: " CLASS_BEFORE = ("Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife , kesinti . " "Yanit olarak tek etiket yaz . Talep: ") CLASS_STEP = ("Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife , kesinti . " "Once adim adim incele , sonra yalniz etiketi yaz . Talep: ") x = CASES[0] print(f"FICTIONAL calculation set {len(CASES)} records, FICTIONAL ticket set {len(TICKETS)} records") print(f"example {x['id']}: {x['text']} , correct answer {exact_answer(x['m3'], x['type'])}") for name, prompt, u in (("previous", CALC_BEFORE, MAX_OUT), ("stepwise", CALC_STEP, MAX_OUT_STEPS)): s = call(prompt + x["text"], max_out=u) s2 = call(prompt + x["text"], seed=SEED + 1, max_out=u) print(f"\n{name} prompt, input {s['input']} output {s['output']} tokens: {s['response']}") print(f" same prompt seed {SEED + 1}: {read_number(s2['response'])}") t = TICKETS[2] print(f"\nclassification example {t['id']} , correct label {t['topic']}:") for name, prompt, u in (("previous", CLASS_BEFORE, MAX_OUT), ("stepwise", CLASS_STEP, MAX_OUT_STEPS)): s = call(prompt + t["text"], max_out=u) print(f" {name} prompt , {s['input']}+{s['output']} tokens: {s['response']}") ``` ``` FICTIONAL calculation set 60 records, FICTIONAL ticket set 120 records example 7001: tuketim 14 m3 abone tipi resmi tutar ne kadar , correct answer 63 previous prompt, input 71 output 2 tokens: sonuc 62 same prompt seed 20260219: 70 stepwise prompt, input 77 output 43 tokens: basamak 1 , 10 m3 , birim 3 , ara toplam 30 . basamak 2 , 4 m3 , birim 5 , ara toplam 50 . sabit bedel 20 ara toplam 70 . indirim 10 sonuc 63 same prompt seed 20260219: 63 classification example 4003 , correct label ariza: previous prompt , 50+1 tokens: ariza stepwise prompt , 55+33 tokens: birinci adim : talep metnindeki isaretler tarandi . ikinci adim : en guclu isaret ariza kavramini gosteriyor . ucuncu adim : etiket ariza ``` The one-move response said **62**, the correct answer is **63**; on the second seed it said **70**. Both are wrong and there is no intermediate step in between. The stepwise response wrote the tiers one by one: the first 10 m³ at unit 3 gave 30, the remaining 4 m³ at unit 5 gave 20, with the fixed fee 70, and after the discount is deducted **63** — the same on both seeds. The cost is on the same line: the output tokens went **from 2 to 43**. The last two lines are the same change on the other task. In classification, asking for step by step did not change the label, but the output tokens went **from 1 to 33**. The thirty-three-token rationale carries no information that the one-token response does not. #### The Same Change on Two Tasks The measurement is repeated on the whole set. Each prompt is run on three seeds and the number of calls is equal on both paths. ```python def measure(dataset, prompt, seed, max_out, correct, read_response): COUNTER[0] = 0 n_in = n_out = hits = 0 for x in dataset: s = call(prompt + x["text"], seed=seed, max_out=max_out) n_in, n_out, hits = n_in + s["input"], n_out + s["output"], hits + (read_response(s["response"]) == correct(x)) return {"accuracy": hits / len(dataset), "input": n_in, "output": n_out, "total": n_in + n_out} def report(title, dataset, prompt1, prompt2, correct, read_response): A = [measure(dataset, prompt1, SEED + j, MAX_OUT, correct, read_response) for j in range(3)] B = [measure(dataset, prompt2, SEED + j, MAX_OUT_STEPS, correct, read_response) for j in range(3)] print(f"{title:<26} {'previous':>9} {'stepwise':>9} {'diff':>9}") for j in range(3): print(f"{'accuracy seed ' + str(SEED + j):<26} {A[j]['accuracy']:>9.4f} " f"{B[j]['accuracy']:>9.4f} {B[j]['accuracy'] - A[j]['accuracy']:>+9.4f}") band = [max(x['accuracy'] for x in v) - min(x['accuracy'] for x in v) for v in (A, B)] print(f"{'three-run band':<26} {band[0]:>9.4f} {band[1]:>9.4f}") for name, k in (("input tokens", "input"), ("output tokens", "output"), ("total tokens", "total")): print(f"{name:<26} {A[0][k]:>9d} {B[0][k]:>9d} {B[0][k] - A[0][k]:>+9d}") print(f"tokens {B[0]['total'] / A[0]['total']:.2f}x, per correct response " f"{A[0]['total'] / A[0]['accuracy'] / len(dataset):.1f} -> " f"{B[0]['total'] / B[0]['accuracy'] / len(dataset):.1f} tokens\n") report("multi-step calculation", CASES, CALC_BEFORE, CALC_STEP, lambda x: exact_answer(x["m3"], x["type"]), read_number) report("single-step classification", TICKETS, CLASS_BEFORE, CLASS_STEP, lambda x: x["topic"], read_label) ``` ``` multi-step calculation previous stepwise diff accuracy seed 20260218 0.3000 0.8833 +0.5833 accuracy seed 20260219 0.3500 0.8833 +0.5333 accuracy seed 20260220 0.3000 0.9000 +0.6000 three-run band 0.0500 0.0167 input tokens 4260 4620 +360 output tokens 118 3334 +3216 total tokens 4378 7954 +3576 tokens 1.82x, per correct response 243.2 -> 150.1 tokens single-step classification previous stepwise diff accuracy seed 20260218 0.7750 0.7250 -0.0500 accuracy seed 20260219 0.7000 0.6583 -0.0417 accuracy seed 20260220 0.7333 0.8083 +0.0750 three-run band 0.0750 0.1500 input tokens 5300 5900 +600 output tokens 192 4023 +3831 total tokens 5492 9923 +4431 tokens 1.81x, per correct response 59.1 -> 114.1 tokens ``` The upper table is where the technique works. Accuracy rises **from 0.3000 to 0.8833** and the difference is large with the same sign on all three runs: +0.5833, +0.5333, +0.6000. The run bands are **0.0500** and **0.0167**; the difference is more than ten times the band, so it counts as **measured**. Total tokens grow **1.82 times**, almost all of the increase coming from the output; on the other hand the cost per correct response falls **from 243.2 to 150.1 tokens**. The lower table is the task where the same technique does not pay. The difference is **-0.0500**, **-0.0417** and **+0.0750**; **not even the sign is stable**, and all three stay inside the bands of 0.0750 and 0.1500. By the rule of the course this difference is **unmeasured**: the wobble between runs of the same prompt is larger than the wobble the change adds. On the cost side there is no uncertainty: total tokens grow **1.81 times** and the cost per correct response rises **from 59.1 to 114.1 tokens**. #### Where the Gain Comes From What creates the difference is the structure of the task. The tariff calculation is chained; writing the intermediate steps breaks the chain into pieces and the error of each piece stays small. Classification is single-step: the cues are read and the label is chosen. There is no intermediate step to write; the rationale text does not produce the decision, it **narrates** it. The cost, on the other hand, is paid the same on both tasks, because a token does not know whether the intermediate step helped. The rule of the course can be read from here: **a prompt changed without measuring is not an improvement but a guess.** #### Summary - Chain of thought means asking for the **intermediate steps** that lead to the result: a one-sentence prompt change, with the model and the set held fixed. - On the multi-step fictional tariff calculation accuracy rises **from 0.3000 to 0.8833**; the difference is large with the same sign on all three runs and well outside the run band (0.0500 / 0.0167). - On single-step classification the differences are **-0.0500**, **-0.0417** and **+0.0750**; the sign changes and they stay inside the run band (0.0750 / 0.1500). **This difference counts as unmeasured.** - The cost is certain on both tasks: total tokens grow **1.82** and **1.81 times**. The cost per correct response falls **from 243.2 to 150.1** in calculation and rises **from 59.1 to 114.1** in classification — the same technique makes one task cheaper and the other more expensive. #### Next Step Step-by-step solving paid off in the calculation, but in a single run it produced **a single chain**. The chain itself is uncertain too: the accuracy of the stepwise path wobbles between 0.8833 and 0.9000, and with a single chain in hand there is no way to know which one is right. The next lesson turns this into scaffolding: the same prompt is run many times and the responses are voted on. What will be measured is how the gain changes with the number of samples, **how many times the number of calls grows**, and the number of cases where the consensus is reached on **a wrong answer**. ### Self-Consistency Source: https://academia.sh/en/courses/prompt-engineering/self-consistency The gain and the cost of running the same prompt several times and voting on the responses: as the number of samples goes from one to nine, accuracy rises from 0.7667 to 0.8833 and stops after seven, the number of calls goes from 60 to 540 and the tokens per correct response from 172.9 to 1343.6. At nine samples a consensus forms on 58 records, but on five of them the consensus is on a wrong answer; the consumption of those five records lies between 41 and 45 m³, that is, where the error is systematic rather than scattered. The previous lesson showed that asking for a step-by-step solution works on the multi-step calculation: accuracy rose from 0.3000 to 0.8833. But that number came from **a single run**. The model is an uncertain component; the same prompt produces another chain on a second run, and if the chain slips somewhere the result changes. With a single chain in hand, nothing can say whether it is right. Self-consistency turns this uncertainty into a lever: the same prompt is run several times and the responses that come out are **voted on**. In this lesson **the text of the prompt never changes**; the only thing that changes is how many times the same prompt is run. This is why the cost is written with the **number of calls** alongside the token. Majority voting itself was measured in the ensemble methods lesson of the previous curriculum and is not repeated here; what is added is the cost and **the place where the consensus breaks**. - **RT11.** The code does not call a real endpoint; it calls a language model **simulator** written with the standard library. Seed **20260218**. This lesson uses only the calculation path. - **RT12.** The task is the previous lesson's: a 60-record **fictional** tariff calculation, tiers of 10, 25 and 40 m³, with a fixed fee and a subscriber type discount on top. The criterion is **exact equality**. - **RT13.** The prompt is the previous lesson's stepwise prompt and stays the same letter for letter. - **RT14.** On a real endpoint the diversity among samples comes from a **distribution temperature** above zero; in the simulator the seed is taken one higher for each sample. - **RT15.** The numbers of samples are 1, 3, 5, 7 and 9; it **does not go beyond nine**. - **RT16.** In the vote the answer with the most votes is chosen, and on a tie the answer that got there first wins. **A call that hits the rate limit cannot vote**, but it enters the number of calls and the token cost. - **RT17.** **Consensus**: there is consensus if more than half of the valid votes are on the same answer. With a single sample the consensus always forms and says nothing. - **RT18.** The simulator's step slip is of two kinds: a **0.05 scattered slip** at every step, and a **0.85 systematic slip** if the remainder falling on the last tier is 5 m³ or less. The second is a modeling assumption and is the source of what this lesson measures. - **RT19.** The whole measurement is repeated with two seed bases: **20260218** and **20260258**. - **RT20.** The cost is three numbers: total tokens, the **number of calls** and the tokens per correct response. #### Nine Samples of the Same Prompt First, what happens on a single record is examined. The same prompt is run nine times, the seed goes up by one in each run, and the last number in the response counts as one vote. Two records were chosen: one ordinary, the other just above the 40 m³ limit. ```python # SIMULATOR. This code does not call a real model endpoint; it is a language model # simulator written with the standard library. The tickets and the tariff are FICTIONAL too. SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUT_STEPS = 240, 96 # context window and the output limit for the stepwise response def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): # FICTIONAL tokenizer: a piece has at most five characters out = [] for piece in text.lower().replace(",", " , ").split(): while len(piece) > 5: out.append(piece[:5]) piece = piece[5:] out.append(piece) return out def trace_id(text): # depends on the prompt's content, independent of the run h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h # FICTIONAL tariff: tier upper bound (m3) and unit price, fixed fee, subscriber type discount TIERS, FIXED_FEE = [(10, 3), (25, 5), (40, 8), (10 ** 6, 12)], 20 SUBSCRIBER_TYPES = {"konut": 0, "tarim": 20, "resmi": 10} CASES = [] for i in range(60): r = rng(SEED + 900 + i) m3 = 6 + int(r() * 52) sub_type = sorted(SUBSCRIBER_TYPES)[pick(r(), [0.40, 0.34, 0.26])] CASES.append({"id": 7001 + i, "m3": m3, "type": sub_type, "text": f"tuketim {m3} m3 abone tipi {sub_type} tutar ne kadar"}) def tiered_total(m3): # total of the consumption split across tiers low, left, total = 0, m3, 0 for high, unit in TIERS: share = min(left, high - low) total, left, low = total + share * unit, left - share, high if left <= 0: break return total def exact_answer(m3, sub_type): # the task's correct answer, outside the simulator return ((tiered_total(m3) + FIXED_FEE) * (100 - SUBSCRIBER_TYPES[sub_type])) // 100 COUNTER = [0] def _parse_task(toks): # consumption and subscriber type from the ticket at the end of the prompt return (next((int(u) for u in reversed(toks) if u.isdigit()), 0), next((t for t in reversed(toks) if t in SUBSCRIBER_TYPES), "konut")) def _step_by_step(m3, sub_type, r): # each tier written separately, a slip margin per step low, left, total, steps = 0, m3, 0, [] for i, (high, unit) in enumerate(TIERS): share = min(left, high - low) # if the remainder in the last tier is small the slip is systematic: it always goes to the same wrong value u = unit if r() < (0.85 if i == 3 and share <= 5 else 0.05): u = TIERS[i - 1][1] if i + 1 == len(TIERS) else TIERS[i + 1][1] total, left, low = total + share * u, left - share, high steps.append(f"basamak {i + 1} , {share} m3 , birim {u} , ara toplam {total}") if left <= 0: break if r() < 0.05: steps.append(f"sabit bedel atlandi ara toplam {total}") else: total += FIXED_FEE steps.append(f"sabit bedel {FIXED_FEE} ara toplam {total}") discount = 0 if r() < 0.05 else SUBSCRIBER_TYPES[sub_type] total = (total * (100 - discount)) // 100 return " . ".join(steps + [f"indirim {discount} sonuc {total}"]) def call(prompt, seed=SEED, max_out=MAX_OUT_STEPS): COUNTER[0] += 1 toks = tokenize(prompt) if len(toks) > WINDOW: return {"status": "context_overflow", "input": len(toks), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: # the rate limit depends on the number of calls return {"status": "rate_limit", "input": len(toks), "output": 0, "response": ""} m3, sub_type = _parse_task(toks) y = _step_by_step(m3, sub_type, rng(seed ^ trace_id(prompt))) out = tokenize(y) if len(out) > max_out: # stop condition out = out[:max_out] y = " ".join(out) return {"status": "ok", "input": len(toks), "output": len(out), "response": y} def read_number(y): # the last number in the response return next((int(x) for x in reversed(y.split()) if x.isdigit()), None) PROMPT = ("Tarife: ilk 10 m3 birim 3 , sonraki 15 m3 birim 5 , sonraki 15 m3 birim 8 , " "40 ustu birim 12 . Sabit bedel 20 . Abone tipi indirimi yuzde konut 0 , " "tarim 20 , resmi 10 . Tutari hesapla . Once her basamagi ayri ayri adim adim " "yaz , sonra sonucu yaz . Talep: ") def sample_vote(x, n, base): # the same prompt is run n times, the responses are voted on votes = {} for j in range(n): v = read_number(call(PROMPT + x["text"], seed=base + j)["response"]) if v is not None: # a call that hits the rate limit does not vote votes[v] = votes.get(v, 0) + 1 if not votes: return None, 0 top = max(votes.values()) return next(k for k in votes if votes[k] == top), top COUNTER[0] = 0 for x in (CASES[3], next(c for c in CASES if 40 < c["m3"] <= 45)): votes, correct = {}, exact_answer(x["m3"], x["type"]) for j in range(9): v = read_number(call(PROMPT + x["text"], seed=SEED + j)["response"]) votes[v] = votes.get(v, 0) + 1 chosen, top = sample_vote(x, 9, SEED) print(f"record {x['id']}: {x['m3']} m3 , {x['type']} , correct answer {correct}") print(f" nine samples: {sorted(votes.items(), key=lambda p: -p[1])}") print(f" consensus {chosen} ({top}/9) -> {'correct' if chosen == correct else 'WRONG'}") ``` ``` record 7004: 12 m3 , resmi , correct answer 54 nine samples: [(54, 6), (36, 2), (59, 1)] consensus 54 (6/9) -> correct record 7008: 45 m3 , tarim , correct answer 244 nine samples: [(228, 5), (276, 2), (264, 1), (244, 1)] consensus 228 (5/9) -> WRONG ``` On the first record voting works. Six of the nine samples say **54**, and that is the correct answer. The remaining three votes are spread over two different wrong answers: **36**, the response where the fixed fee was skipped; **59**, the response where the unit of the second tier slipped. The errors differ from one another, while the correct answer is the same in all of them. This is why scattered error is fixed by voting — there are many ways to go wrong, and one way to go right. On the second record the same scaffolding works in reverse. The correct answer is **244**, and only **one** of the nine samples says it. The majority, **five** of the nine, gathers on **228**: the last 5 m³ of the 45 m³ was calculated with the unit of one tier lower rather than the unit of the top tier. Voting does not fix this record, it **reinforces the error**. The difference lies in the type of the error: the first is scattered, the second systematic. #### Number of Samples, Gain and Calls The same measurement is repeated on the whole set and at five different numbers of samples. Each row is run with two seed bases; the second column is the second run of the same scaffolding. ```python def measure(n, base): COUNTER[0] = 0 n_in = n_out = correct = agreed = empty = 0 broken = [] for x in CASES: votes, answer = {}, exact_answer(x["m3"], x["type"]) for j in range(n): s = call(PROMPT + x["text"], seed=base + j) n_in, n_out = n_in + s["input"], n_out + s["output"] v = read_number(s["response"]) if v is None: empty += 1 # a call that hits the rate limit does not vote else: votes[v] = votes.get(v, 0) + 1 if not votes: continue # the single sample hit the rate limit top = max(votes.values()) chosen = next(k for k in votes if votes[k] == top) correct += chosen == answer if top * 2 > sum(votes.values()): # more than half on the same answer agreed += 1 if chosen != answer: broken.append(x["m3"]) return {"accuracy": correct / len(CASES), "calls": COUNTER[0], "tokens": n_in + n_out, "agreed": agreed, "broken": broken, "empty": empty} print(f"{'samples':>7} {'accuracy':>8} {'second':>8} {'calls':>6} {'tokens':>9} " f"{'per_correct':>11} {'agreed':>8} {'on_wrong':>9}") for n in (1, 3, 5, 7, 9): A, B = measure(n, SEED), measure(n, SEED + 40) print(f"{n:>7} {A['accuracy']:>8.4f} {B['accuracy']:>8.4f} {A['calls']:>6} " f"{A['tokens']:>9} {A['tokens'] / A['accuracy'] / len(CASES):>11.1f} " f"{A['agreed']:>8} {len(A['broken']):>9}") if n == 9: print(f"\nconsumption of the records with a wrong consensus at nine samples: " f"{sorted(A['broken'], reverse=True)} m3") print(f"calls that hit the rate limit and could not vote: {A['empty']}") ``` ``` samples accuracy second calls tokens per_correct agreed on_wrong 1 0.7667 0.6667 60 7954 172.9 59 13 3 0.8500 0.8667 180 23785 466.4 56 6 5 0.8667 0.8833 300 39552 760.6 55 4 7 0.8833 0.9000 420 55389 1045.1 57 5 9 0.8833 0.8833 540 71211 1343.6 58 5 consumption of the records with a wrong consensus at nine samples: [45, 45, 44, 43, 41] m3 calls that hit the rate limit and could not vote: 14 ``` The first column gives the gain. With one sample accuracy is **0.7667**, with nine samples **0.8833**; the gain is **+0.1167**. The distribution of the gain is not even: the first three samples bring **+0.0833**, the next four samples only **+0.0333**, and the move from seven to nine brings **nothing**. The gain saturates at the seventh sample. The second column is the most important number of this lesson. With one sample the two runs give **0.7667** and **0.6667**; the difference between them is **0.1000**. With nine samples both runs give **0.8833**; the difference is **0.0000**. The one-sample gain (+0.1167) stays only a little above the run difference of one sample (0.1000), so it would be hard to count it as measured by looking at the first run. On the second run the same gain is **+0.2166**. The most robustly measured effect of the technique is not that it raises accuracy, but that it **brings the wobble from run to run down to zero**. The third and fourth columns are the cost, and it is not debatable. The number of calls goes **from 60 to 540**, total tokens **from 7954 to 71211** — almost exactly nine times. The cost per correct response rises **from 172.9 to 1343.6 tokens**, that is, **7.8 times**. A cost nine times larger while the gain stays at 0.1167 says in a single line why this technique cannot be switched on for every task. The number for three samples (0.8500, 466.4 tokens) is in most cases a more defensible point than nine samples. #### Where the Consensus Breaks The last two columns look at the voting itself. With one sample **there is consensus on 59 records** and **13 of them are wrong**, because a single vote is always the majority. This shows that consensus alone says nothing. With nine samples the records with consensus are **58**, and the ones with consensus on a wrong answer **5**. The number of records with consensus on a wrong answer, as the number of samples goes from three to nine, **gets stuck at 6, 4, 5 and 5**. The consumption of these five records is printed: **45, 45, 44, 43 and 41 m³**. All of them are just above the 40 m³ limit. In these records the remainder falling on the last tier is small, and the simulator calculates it systematically from the unit of one tier lower. The error is not random, which is why all nine samples go to the same wrong answer. **Self-consistency fixes scattered error and reinforces systematic error.** Consensus is not proof of correctness; it is only a sign that the error is not scattered. There is one more place where the cost does not turn into gain: **14** of the 540 calls hit the rate limit, could not vote, and still entered the number of calls. Every scaffolding that multiplies the number of calls by nine also multiplies the surface's failure share by nine. #### Summary - Self-consistency means running **the same prompt** several times and voting on the responses; the text of the prompt does not change, the number of runs does. - As the number of samples goes from one to nine, accuracy rises **from 0.7667 to 0.8833**; **+0.0833** of the gain comes **from the first three samples** and the gain stops after the seventh sample. - The cost is written in two units: calls **from 60 to 540**, total tokens **from 7954 to 71211**, tokens per correct response **from 172.9 to 1343.6**. Tokens nine times, gain 0.1167. - The most robustly measured effect of the technique is that it brings the run difference down **from 0.1000 to 0.0000**; the gain of a single sample stays only a little above the run difference of a single sample. - With nine samples a consensus forms on 58 records, and on **5 of them the consensus is on a wrong answer**. The consumption of these five records is between 41 and 45 m³: when the error is systematic, voting does not fix it, it reinforces it. #### Next Step Generating several solutions and voting worked, but the generated solutions had one thing in common: each was built **independently from start to finish**. Nine samples started from zero nine times, none saw where the other slipped, and none took over the good first half of another. **Branching from the middle** of a solution was never tried. The next lesson tries it: the solution stops being a single chain and becomes a structure that grows step by step and splits in several directions at every step. What will be measured is the branching factor and **how the number of calls grows** as the depth increases, and where the gain stops. ### Tree of Thoughts Source: https://academia.sh/en/courses/prompt-engineering/tree-of-thought Measuring branching from the middle of a solution: on the same 48 fictional support tickets, the arrangement that asks for all thirty-two fault codes in a single prompt gives 0.4792 accuracy with 48 calls; the tree of thoughts, which splits the solution into steps and asks for several extensions at every step, falls to 0.0833 at branch 1, rises to 0.6250 at branch 3 and stops at branch 4. As branching goes from 3 to 4, accuracy slips by 0.0208 while the number of calls goes from 795 to 1086 and tokens from 43182 to 59835; since this difference is smaller than the 0.0417 difference between two runs of the same arrangement, it counts as unmeasured. The previous lesson ran the same prompt several times and voted on the responses. Self-consistency worked: the number a single response caught went up with the consensus of nine samples. But the generated solutions had one thing in common. Each was generated **independently from start to finish**; none saw where another went wrong. The first step of a solution was never held fixed while the second step was tried in several different directions, and a path going badly was never abandoned halfway. This lesson fills that gap. What is changed is again the text, but this time not a single prompt but **the arrangement of prompts**: instead of one long prompt, short prompts that each move the solution one step forward, and a rule that selects the branches to keep. What will be measured is what this gains and **how many calls** it costs. - **RT21.** The task is **fictional**: short support tickets are assigned a fault code with three components — area, symptom and degree. Four areas, four symptoms per area, two degrees; thirty-two codes in total. - **RT22.** What is called is a **simulator**; no request goes to a real model endpoint. The simulator keeps the form built in this curriculum's introductory course, with one difference: the option set is given from outside and each option is a **path**, whose score is summed from the cues of its components. - **RT23.** Seed **20260218**; the same prompt gives the same response on the same seed. Uncertainty is measured by **changing the seed**, and the second run of every arrangement is taken with 20260219. - **RT24.** In the ticket text the area cue is **weak** and the symptom cue is **strong**; also, every ticket mentions **a misleading area** whose cue is heavier than the cue of the correct area. This construction opens the place where the tree will be measured: the first step is taken with the weakest evidence. - **RT25.** The tree makes two kinds of calls. An **extension** call extends a path by one step and is repeated as many times as the number of branches; a **selection** call filters the candidates that accumulate down to the ones to keep. Both are counted. - **RT26.** The branching factor and the number of kept branches are held **equal**. The scale is small: at most four branches, at most three levels of depth. The criterion is **all three components being correct at once**. - **RT27.** The cost is written in three units: the **number of calls**, **tokens** and **failed calls**. Since the rate limit depends on the number of calls, branching multiplies failures too. #### Branching from the Middle of a Path The previous prompt is a single one and asks for everything at once: all thirty-two codes are written into the prompt, the ticket is added, a single response is expected. The changed arrangement splits the same job into three steps — first the **area**, then the **symptom** of that area, then the **degree** — and asks for several separate extensions at every step; a part of the accumulated candidates is kept, the rest is dropped. The first block builds the simulator, the fictional code tree and the first two levels of one ticket. ```python # SIMULATOR. This code does not call a real model endpoint; it is a language model # simulator written with the standard library. The tickets and the fault codes are FICTIONAL too. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW = 240 # context window, tokens def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): # FICTIONAL tokenizer: a piece has at most five characters out = [] for piece in text.lower().replace(",", " , ").split(): while len(piece) > 5: out.append(piece[:5]) piece = piece[5:] out.append(piece) return out def trace_id(text): # depends on the prompt's content, independent of the run h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h AREAS = ["dagitim", "sayac", "abonelik", "saha"] SYMPTOMS = {"dagitim": ["kesinti", "basinc", "sizinti", "tikanma"], "sayac": ["endeks", "bozukluk", "degisim", "okuma"], "abonelik": ["tarife", "fatura", "odeme", "kayit"], "saha": ["randevu", "erisim", "malzeme", "gecikme"]} DEGREES = ["birinci", "ikinci"] AREA_WORD = {"dagitim": "sebeke", "sayac": "olcum", "abonelik": "sozlesme", "saha": "ekip"} TRAP_WORD = {"dagitim": "hatti", "sayac": "cihaz", "abonelik": "hesap", "saha": "servis"} SYMPTOM_WORD = {"kesinti": "akmiyor", "basinc": "zayif", "sizinti": "islak", "tikanma": "dolmus", "endeks": "rakam", "bozukluk": "calismiyor", "degisim": "yenisi", "okuma": "girilmemis", "tarife": "basamak", "fatura": "tutar", "odeme": "borcum", "kayit": "musteri", "randevu": "saatte", "erisim": "kapali", "malzeme": "parca", "gecikme": "bekliyor"} DEGREE_WORD = {"birinci": "acil", "ikinci": "normal"} AREA_CUE = {AREA_WORD[a][:5]: a for a in AREAS} # area cue, weak TRAP_CUE = {TRAP_WORD[a][:5]: a for a in AREAS} # misleading area cue, heavier SYMPTOM_CUE = {SYMPTOM_WORD[s][:5]: s for a in AREAS for s in SYMPTOMS[a]} DEGREE_CUE = {DEGREE_WORD[d][:5]: d for d in DEGREES} TICKETS = [] for i in range(48): r = rng(SEED + i) area = AREAS[int(r() * 4)] symptom = SYMPTOMS[area][int(r() * 4)] degree = DEGREES[int(r() * 2)] trap = AREAS[(AREAS.index(area) + 1 + int(r() * 3)) % 4] # the misleading area in the ticket if r() < 0.5: text = (f"{TRAP_WORD[trap]} icin kayit acilmisti ama sorun {AREA_WORD[area]} tarafinda , " f"{SYMPTOM_WORD[symptom]} durumu var , {DEGREE_WORD[degree]} bakilmasi gerekiyor") else: text = (f"{TRAP_WORD[trap]} uzerinden bildirdim , {AREA_WORD[area]} ile ilgili " f"{SYMPTOM_WORD[symptom]} diye yaziyor , {DEGREE_WORD[degree]} olarak degerlendirin") TICKETS.append({"id": 5001 + i, "text": text, "code": (area, symptom, degree)}) ALL_CODES = [(a, s, d) for a in AREAS for s in SYMPTOMS[a] for d in DEGREES] COUNTER = [0, 0, 0] # number of calls, tokens, failed calls def join_path(p): return " ".join(p) def extend_paths(path, k): # all options that extend the path by k components paths = [path] for _ in range(k): grown = [] for q in paths: grown += [q + (s,) for s in (AREAS if not q else SYMPTOMS[q[0]] if len(q) == 1 else DEGREES)] paths = grown return paths def call(prompt, options, seed=SEED, temperature=0.6): COUNTER[0] += 1 toks = tokenize(prompt) COUNTER[1] += len(toks) if len(toks) > WINDOW or COUNTER[0] % 37 == 0: COUNTER[2] += 1 return {"input": len(toks), "response": None} r = rng(seed ^ trace_id(prompt)) p = {k: 0.15 for k in options} for t in toks: for table, w in ((AREA_CUE, 1.0), (TRAP_CUE, 1.3), (SYMPTOM_CUE, 1.6), (DEGREE_CUE, 0.9)): h = table.get(t) if h is not None: for k in p: if h in k: # an option is a path; the score is summed from its components p[k] += w d = sorted(p, key=join_path) if temperature <= 0.0: k = max(d, key=lambda a: p[a]) else: e = [math.exp(p[a] / temperature) for a in d] k = d[pick(r(), [v / sum(e) for v in e])] COUNTER[1] += len(tokenize(join_path(k))) return {"input": len(toks), "response": k} def build_prompt(lead, options, text): return (lead + " secenekler : " + " , ".join(join_path(k) for k in options) + " . Talep : " + text) x = TICKETS[2] print(f"FICTIONAL: {len(TICKETS)} tickets, {len(ALL_CODES)} fault codes " f"(4 areas x 4 symptoms x 2 degrees)") print(f"ticket {x['id']}: {x['text']}") print(f"correct code: {join_path(x['code'])}") s = call(build_prompt("Talebe bir ariza kodu ata .", ALL_CODES, x["text"]), ALL_CODES) print(f"single call: input {s['input']} tokens, response '{join_path(s['response'])}'") first, candidates = extend_paths((), 1), [] print("\nthree separate extension calls at the first level:") for j in range(3): u = call(build_prompt("Cozum yolunu bir adim uzat .", first, x["text"]), first, SEED + j) print(f" candidate {j + 1}: {join_path(u['response'])} (input {u['input']} tokens)") if u["response"] not in candidates: candidates.append(u["response"]) print("the extension of each branch at the second level:") for p in candidates: se = extend_paths(p, 1) u = call(build_prompt("Cozum yolunu bir adim uzat .", se, x["text"]), se) print(f" {join_path(p)} -> {join_path(u['response'])}") ``` ``` FICTIONAL: 48 tickets, 32 fault codes (4 areas x 4 symptoms x 2 degrees) ticket 5003: hesap uzerinden bildirdim , olcum ile ilgili rakam diye yaziyor , acil olarak degerlendirin correct code: sayac endeks birinci single call: input 235 tokens, response 'sayac endeks birinci' three separate extension calls at the first level: candidate 1: abonelik (input 43 tokens) candidate 2: abonelik (input 43 tokens) candidate 3: sayac (input 43 tokens) the extension of each branch at the second level: abonelik -> abonelik kayit sayac -> sayac endeks ``` Two numbers are visible right away. Writing the thirty-two codes into the prompt raises the prompt to **235 tokens**; the context window is 240, so the single-prompt arrangement stands five tokens below the limit. Since the tree's prompt carries only the options of that level, it is **43 tokens**. This is the first gain of splitting. The bottom four lines show why branching is needed. The word "hesap" (account) in the ticket is the cue of the misleading area and is heavier than the word "olcum" (measurement), the cue of the correct area; so the first two extension calls say `abonelik` (billing), and the correct area `sayac` (meter) appears only on the third. A single-branch search would end here and, producing `abonelik kayit` at the second level, would go deeper on a path with no counterpart in the ticket. When the correct branch is kept, the second level gives `sayac endeks`, because the symptom cue is strong. **This is what the tree promises: a wrong first step taken with weak evidence can be taken back, as long as it is not the only step.** #### Branch, Depth and Number of Calls The second block builds the tree and puts six arrangements side by side: branching is raised to one, two, three and four, then branch is held at three and depth is lowered to one and two. The single call and the previous lesson's five-sample self-consistency also enter the table; the second run of every row is taken by changing the seed. ```python SPLIT = {1: (3,), 2: (2, 1), 3: (1, 1, 1)} # how many steps the path is split into def tree(x, branch, keep, depth, seed): paths = [()] for k in SPLIT[depth]: candidates = [] for p in paths: # generation: extend each branch `branch` times se = extend_paths(p, k) for j in range(min(branch, len(se))): s = call(build_prompt("Cozum yolunu bir adim uzat .", se, x), se, seed + j) if s["response"] is not None and s["response"] not in candidates: candidates.append(s["response"]) if not candidates: candidates = extend_paths(paths[0], k)[:1] if len(candidates) <= keep: paths = candidates # nothing to choose, no call is spent else: paths = [] for j in range(keep): # selection: filter the branches to keep s = call(build_prompt("En umut verici yollari sec .", candidates, x), candidates, seed + j) if s["response"] is not None and s["response"] not in paths: paths.append(s["response"]) paths = paths or candidates[:keep] s = call(build_prompt("Kalan yollardan birini kod olarak yaz .", paths, x), paths, seed) return s["response"] or paths[0] def single(x, k, seed): # k = 1 single call, k > 1 self-consistency counts = {} for j in range(k): s = call(build_prompt("Talebe bir ariza kodu ata .", ALL_CODES, x), ALL_CODES, seed + j) if s["response"] is not None: counts[s["response"]] = counts.get(s["response"], 0) + 1 return max(sorted(counts, key=join_path), key=lambda a: counts[a]) if counts else None def measure(f, seed): COUNTER[:] = [0, 0, 0] correct = 0 for x in TICKETS: correct += f(x["text"], seed) == x["code"] return {"accuracy": correct / len(TICKETS), "calls": COUNTER[0], "tok": COUNTER[1], "failed": COUNTER[2]} ARRANGEMENTS = [("single call", lambda x, t: single(x, 1, t)), ("self-consistency 5 samples", lambda x, t: single(x, 5, t)), ("tree branch 1 depth 3", lambda x, t: tree(x, 1, 1, 3, t)), ("tree branch 2 depth 3", lambda x, t: tree(x, 2, 2, 3, t)), ("tree branch 3 depth 3", lambda x, t: tree(x, 3, 3, 3, t)), ("tree branch 4 depth 3", lambda x, t: tree(x, 4, 4, 3, t)), ("tree branch 3 depth 1", lambda x, t: tree(x, 3, 3, 1, t)), ("tree branch 3 depth 2", lambda x, t: tree(x, 3, 3, 2, t))] print(f"{'prompt arrangement':<28}{'accuracy':>8}{'run2':>8}{'diff':>8}{'calls':>7}" f"{'tokens':>10}{'failed':>10}") for name, f in ARRANGEMENTS: A, B = measure(f, SEED), measure(f, SEED + 1) print(f"{name:<28}{A['accuracy']:>8.4f}{B['accuracy']:>8.4f}" f"{B['accuracy'] - A['accuracy']:>+8.4f}{A['calls']:>7d}{A['tok']:>10d}" f"{A['failed']:>10d}") ``` ``` prompt arrangement accuracy run2 diff calls tokens failed single call 0.4792 0.5208 +0.0417 48 11617 1 self-consistency 5 samples 0.7292 0.7708 +0.0417 240 58074 6 tree branch 1 depth 3 0.0833 0.2708 +0.1875 192 9776 5 tree branch 2 depth 3 0.4167 0.5417 +0.1250 506 26651 13 tree branch 3 depth 3 0.6250 0.5833 -0.0417 795 43182 21 tree branch 4 depth 3 0.6042 0.5833 -0.0208 1086 59835 29 tree branch 3 depth 1 0.6250 0.6250 +0.0000 192 37617 5 tree branch 3 depth 2 0.7292 0.7500 +0.0208 421 30650 11 ``` The third row is the harshest number of this lesson. Splitting the solution into three steps but **never branching** drops accuracy from 0.4792 to **0.0833**, and it does this with four times the calls. The reason was seen in the first block: a search that does not branch takes the first step with the weakest evidence and can never come back. In the single-prompt arrangement the three components were evaluated **together at the same score**, so the strong symptom cue could carry the weak area cue; splitting cut that carrying. **Splitting into steps is not an improvement by itself; its cost is certain, its gain is negative.** Branching takes this back. At branch two it is 0.4167, at branch three **0.6250**. Where the gain stops is visible at the fourth step: when branching is raised to four, accuracy **slips by 0.0208**, while the number of calls goes from 795 to **1086**, tokens from 43182 to **59835**, failed calls from 21 to **29**. The difference between two runs of the same arrangement is 0.0417 at branch three. Since the difference that the move from branch three to four contributes is smaller than this number, it **counts as unmeasured**: two hundred ninety-one calls and more than sixteen thousand tokens were spent on a change that cannot be shown. The last two rows are the depth axis and give the opposite result. When depth is lowered from three to two with the same number of branches, accuracy rises from 0.6250 to **0.7292** and calls fall from 795 to 421: when area and symptom are chosen together, the strong symptom cue can again carry the weak area cue. When depth falls to one, accuracy stays at 0.6250 and calls drop to **192**, but because the prompts carry all thirty-two options at once, tokens rise from 30650 to 37617. **The ranking by calls is not the ranking by tokens.** The second row of the table closes the lesson. The previous lesson's five-sample self-consistency gives **0.7292** — the same number as the best tree arrangement — and does it with **240 calls** instead of 421. In tokens the situation reverses: 58074 against 30650. The choice between the two techniques is not a ranking of superiority but **a choice of unit**; the only measurable thing the tree contributes on this task is that it reaches the same accuracy with fewer tokens. **A prompt changed without measuring is not an improvement but a guess**, and in this table half of the changed prompts were not an improvement. #### Summary - The tree of thoughts is not a single prompt but **an arrangement of prompts**: the solution is split into steps, several extensions are asked for at every step, a part of the candidates is kept. Extension and selection calls are counted together. - Splitting into steps without branching does harm: accuracy falls from 0.4792 to **0.0833**, calls go from 48 to 192. When the wrongly taken first step is the only step, it cannot be taken back. - Branching gains but saturates: from branch 1 to 3 accuracy rises **from 0.0833 to 0.6250**, at branch 4 it **slips by 0.0208**. Calls go from 192 to 1086, tokens from 9776 to 59835. - The difference of the move from branch three to four is **smaller than the 0.0417 difference** between two runs of the same arrangement; it counts as unmeasured and cannot be written as a gain. - Depth works in the opposite direction: using 2 instead of 3 raises accuracy to **0.7292** and lowers calls to 421. Self-consistency gives the same accuracy with **240 calls** and **58074 tokens**; the tree's 30650 tokens are cheap, its 421 calls expensive. #### Next Step These three techniques all worked in the same direction: the model was asked for **more steps**. Chain of thought had the steps written inside a single response, self-consistency had the same steps produced and voted on many times, the tree of thoughts put a selection rule between the steps. In all three the steps are **at the same level**: all of them move toward the detail of the solution. What went wrong in this lesson, though, was the detail itself; a misleading word spoiled the first step and all the later steps were built on that spoiled step. **Going up** before going down into the detail was never tried. Asking, before assigning a code to a ticket, which general rule that ticket falls under was an option at hand. The next lesson measures it: the prompt is split into two stages, the principle is asked first and then the solution, and how much the two-stage version moves accuracy compared with the one-stage version is printed side by side with how many tokens it adds. ### Step-Back Prompting Source: https://academia.sh/en/courses/prompt-engineering/step-back-prompting Measuring what it is worth to ask for the principle before solving: on the same 48 fictional support tickets, the single-stage prompt gives 0.5417 code accuracy with 48 calls, while the two-stage prompt, which first asks for the general rule and then asks for the solution, gives 0.7500 and does it with 96 calls and 1.66 times the tokens. The whole gain depends on the first stage: under the right principle code accuracy is 0.8000, and when a wrong principle is imposed it drops to between 0.0000 and 0.1458, that is, below the single-stage prompt. The same arrangement does not move accuracy at all in a task that asks only for the degree, but it raises the tokens to 2.69 times. The previous three lessons always asked the model for the same thing: **more steps**. The steps were written out inside a single response, duplicated and voted on, branched and pruned — and in all three they looked in the same direction, toward the detail of the ticket. In the tree lesson the error came from exactly there: a misleading word spoiled the first step, and no step asked "what should I have been looking at in this ticket". Step-back prompting goes the other way. Before asking for the solution it **raises the level of abstraction**: which general rule do tickets of this kind get solved by. The prompt is still text, but this time **in two stages**, and the output of the first is written into the input of the second. The cost is known up front — the number of calls doubles. What will be measured is what is gained in return. - **RT28.** The task and the fiction are the same as in the previous lesson: 48 fictional support tickets, a fault code with three components, thirty-two codes in total. What is called is a **simulator**; no request goes to a real model endpoint. Seed **20260218**, second run 20260219. - **RT29.** The fictional **principle pool** is four general rules. When a principle is written at the head of the prompt, the weight the simulator gives to each cue changes; this fictional scaffolding is the measurable counterpart of what a real instruction does — an instruction selects evidence. - **RT30.** In this fiction **the correct principle is `gozlem`** (observation): the code is chosen by the symptom and the area the ticket reports, not by the route through which the case was opened. The other three principles are wrong. - **RT31.** The two stages are **two calls**. The response of the first stage is written **verbatim** at the head of the second prompt; there is no correction, verification or fallback in between. - **RT32.** The options are written into the prompt as **component lists**; writing the thirty-two codes one by one leaves no room for the principle. - **RT33.** The cost is written in four units: **number of calls**, **input tokens**, **output tokens** and **failed calls**. The criterion is all three components being correct at once in the code task, and the degree being correct in the degree task. #### Principle First, Then Solution The previous prompt is a single one: instruction, options and ticket go together. The changed prompt is split in two. The first call asks only **which general rule applies**; it sees the ticket but does not see the code options. The second call takes the text of the returned rule at its head and asks for the real job. The first block builds the simulator, the principle pool and the two stages on one ticket. ```python # SIMULATOR. This code does not call a real model endpoint; it is a language model # simulator written with the standard library. The support tickets and the fault codes are FICTIONAL too. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW = 240 # context window, tokens def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): # FICTIONAL tokenizer: a piece has at most five characters out = [] for piece in text.lower().replace(",", " , ").split(): while len(piece) > 5: out.append(piece[:5]) piece = piece[5:] out.append(piece) return out def trace_id(text): # depends on the prompt's content, independent of the run h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h AREAS = ["dagitim", "sayac", "abonelik", "saha"] SYMPTOMS = {"dagitim": ["kesinti", "basinc", "sizinti", "tikanma"], "sayac": ["endeks", "bozukluk", "degisim", "okuma"], "abonelik": ["tarife", "fatura", "odeme", "kayit"], "saha": ["randevu", "erisim", "malzeme", "gecikme"]} DEGREES = ["birinci", "ikinci"] AREA_WORD = {"dagitim": "sebeke", "sayac": "olcum", "abonelik": "sozlesme", "saha": "ekip"} TRAP_WORD = {"dagitim": "hatti", "sayac": "cihaz", "abonelik": "hesap", "saha": "servis"} SYMPTOM_WORD = {"kesinti": "akmiyor", "basinc": "zayif", "sizinti": "islak", "tikanma": "dolmus", "endeks": "rakam", "bozukluk": "calismiyor", "degisim": "yenisi", "okuma": "girilmemis", "tarife": "basamak", "fatura": "tutar", "odeme": "borcum", "kayit": "musteri", "randevu": "saatte", "erisim": "kapali", "malzeme": "parca", "gecikme": "bekliyor"} DEGREE_WORD = {"birinci": "acil", "ikinci": "normal"} AREA_CUE = {AREA_WORD[a][:5]: a for a in AREAS} # area cue, weak TRAP_CUE = {TRAP_WORD[a][:5]: a for a in AREAS} # channel cue, heavier SYMPTOM_CUE = {SYMPTOM_WORD[s][:5]: s for a in AREAS for s in SYMPTOMS[a]} DEGREE_CUE = {DEGREE_WORD[d][:5]: d for d in DEGREES} PRINCIPLE_CUE = {"sorun": "gozlem", "ilgil": "gozlem", "durum": "gozlem", "diye": "gozlem", "taraf": "gozlem", "yaziy": "gozlem", "kayit": "kanal", "uzeri": "kanal", "acilm": "sira", "bildi": "sira", "gerek": "sure", "deger": "sure"} PRINCIPLES = { "gozlem": "ilke gozlem : kod , talebin bildirdigi belirtiye ve yere gore secilir .", "kanal": "ilke kanal : kod , kaydin acildigi kanala gore secilir .", "sure": "ilke sure : kod , bildirimin aciliyet duzeyine gore secilir .", "sira": "ilke sira : kod , talebin geldigi siraya gore secilir ."} RULE = {"gozle": {"A": 2.4, "T": 0.2, "S": 2.8}, # key: first piece of the principle name "kanal": {"A": 0.3, "T": 2.6}, "sure": {"S": 0.6}, "sira": {"A": 0.4, "S": 0.5}} TICKETS = [] for i in range(48): r = rng(SEED + i) area = AREAS[int(r() * 4)] symptom = SYMPTOMS[area][int(r() * 4)] degree = DEGREES[int(r() * 2)] trap = AREAS[(AREAS.index(area) + 1 + int(r() * 3)) % 4] # the channel in the ticket, misleading if r() < 0.5: text = (f"{TRAP_WORD[trap]} icin kayit acilmisti ama sorun {AREA_WORD[area]} tarafinda , " f"{SYMPTOM_WORD[symptom]} durumu var , {DEGREE_WORD[degree]} bakilmasi gerekiyor") else: text = (f"{TRAP_WORD[trap]} uzerinden bildirdim , {AREA_WORD[area]} ile ilgili " f"{SYMPTOM_WORD[symptom]} diye yaziyor , {DEGREE_WORD[degree]} olarak degerlendirin") TICKETS.append({"id": 5001 + i, "text": text, "code": (area, symptom, degree)}) ALL_CODES = [(a, s, d) for a in AREAS for s in SYMPTOMS[a] for d in DEGREES] COUNTER = [0, 0, 0, 0] # calls, input tokens, output tokens, failed calls def join_path(p): return " ".join(p) def call(prompt, options, seed=SEED, temperature=0.6): COUNTER[0] += 1 toks = tokenize(prompt) COUNTER[1] += len(toks) if len(toks) > WINDOW or COUNTER[0] % 37 == 0: COUNTER[3] += 1 return {"input": len(toks), "response": None} weight = {"A": 1.0, "T": 1.3, "S": 1.6, "D": 0.9} for t in toks: weight.update(RULE.get(t, {})) # the principle in the prompt changes the cue weights r = rng(seed ^ trace_id(prompt)) p = {k: 0.15 for k in options} for t in toks: for table, w in ((AREA_CUE, weight["A"]), (TRAP_CUE, weight["T"]), (SYMPTOM_CUE, weight["S"]), (DEGREE_CUE, weight["D"]), (PRINCIPLE_CUE, 1.0)): h = table.get(t) if h is not None: for k in p: if h in k: p[k] += w d = sorted(p, key=join_path) if temperature <= 0.0: k = max(d, key=lambda a: p[a]) else: e = [math.exp(p[a] / temperature) for a in d] k = d[pick(r(), [v / sum(e) for v in e])] COUNTER[2] += len(tokenize(join_path(k))) return {"input": len(toks), "response": k} CODE_PROMPT = ("Talebe bir ariza kodu ata . alanlar : " + " , ".join(AREAS) + " . belirtiler : " + " , ".join(s for a in AREAS for s in SYMPTOMS[a]) + " . dereceler : " + " , ".join(DEGREES) + " . Talep : ") PRINCIPLE_PROMPT = ("Cozmeden once bu talebin hangi genel kurala girdigini yaz . kurallar : " + " , ".join(sorted(PRINCIPLES)) + " . Talep : ") x = TICKETS[2] print(f"FICTIONAL: {len(TICKETS)} tickets, {len(ALL_CODES)} fault codes, " f"{len(PRINCIPLES)} general rules") print(f"ticket {x['id']}: {x['text']}") print(f"correct code: {join_path(x['code'])}") t1 = call(CODE_PROMPT + x["text"], ALL_CODES) print(f"single stage: input {t1['input']} tokens, response '{join_path(t1['response'])}'") i1 = call(PRINCIPLE_PROMPT + x["text"], [(k,) for k in sorted(PRINCIPLES)]) print(f"\nfirst stage: input {i1['input']} tokens, principle '{join_path(i1['response'])}'") for name in ("gozlem", "kanal"): t2 = call(PRINCIPLES[name] + " " + CODE_PROMPT + x["text"], ALL_CODES) print(f"second stage principle {name}: input {t2['input']} tokens, " f"response '{join_path(t2['response'])}'") u = call(PRINCIPLES["gozlem"] + " Talebe bir ariza kodu ata . secenekler : " + " , ".join(join_path(k) for k in ALL_CODES) + " . Talep : " + x["text"], ALL_CODES) print(f"\nif the thirty-two codes were written one by one: input {u['input']} tokens " f"(limit {WINDOW}), response {u['response']}") ``` ``` FICTIONAL: 48 tickets, 32 fault codes, 4 general rules ticket 5003: hesap uzerinden bildirdim , olcum ile ilgili rakam diye yaziyor , acil olarak degerlendirin correct code: sayac endeks birinci single stage: input 100 tokens, response 'abonelik fatura birinci' first stage: input 49 tokens, principle 'gozlem' second stage principle gozlem: input 118 tokens, response 'sayac endeks ikinci' second stage principle kanal: input 115 tokens, response 'abonelik odeme birinci' if the thirty-two codes were written one by one: input 253 tokens (limit 240), response None ``` The single-stage prompt says `abonelik fatura birinci`: the word "hesap" (account) in the ticket is the route through which the case was opened and its cue is heavy; the real evidence, "olcum" (measurement), stayed beneath it. The first stage spends 49 tokens and finds the rule `gozlem` (observation); when the rule is written at the head of the second prompt the response becomes `sayac endeks ikinci` — area and symptom were fixed, the degree is still wrong. The rule says nothing about the degree, and the remaining error sits exactly there. The third line is the other side of the cost. When a wrong principle — `kanal` (route) — is written at the head of the second prompt, the response becomes `abonelik odeme birinci`; worse than the single-stage one. The second stage does not make a correction of its own, it **obeys the rule put in front of it**. The last line is a design constraint: the previous lesson's prompt wrote the thirty-two codes one by one, and when a principle is added at its head the input rises to 253 tokens and exceeds the 240 limit, and the call returns without producing any response. **Step-back prompting needs room to have been opened in the window**; this is why the options were written as a component list. #### What the Two Stages Add and Take The second block measures two tasks: the three-component code and the **degree** alone — the only component written word for word in the ticket. Each task is run single-stage and with a step back, each arrangement in two runs. At the end the source of the gain is separated: in how many tickets the first stage found the right principle, and what the second stage gives when each principle is imposed one by one. ```python DEGREE_PROMPT = "Talebin derecesini yaz . dereceler : " + " , ".join(DEGREES) + " . Talep : " PRINCIPLE_OPTIONS = [(k,) for k in sorted(PRINCIPLES)] CODE, DEGREE = lambda x: x["code"], lambda x: (x["code"][2],) def single(prompt, options): return lambda x, t: (call(prompt + x["text"], options, t)["response"], None) def step_back(prompt, options): # first the principle, then the solution def f(x, t): i = call(PRINCIPLE_PROMPT + x["text"], PRINCIPLE_OPTIONS, t) name = i["response"][0] if i["response"] else "sira" return call(PRINCIPLES[name] + " " + prompt + x["text"], options, t)["response"], name return f def measure(f, target, seed=SEED): COUNTER[:] = [0, 0, 0, 0] hits = right_n = right_hits = wrong_hits = 0 for x in TICKETS: y, name = f(x, seed) ok = y == target(x) hits += ok if name is not None: right_n += name == "gozlem" right_hits += ok and name == "gozlem" wrong_hits += ok and name != "gozlem" n = len(TICKETS) return {"accuracy": hits / n, "n_right": right_n, "right_acc": right_hits / right_n if right_n else 0.0, "wrong_acc": wrong_hits / (n - right_n) if right_n < n else 0.0, "calls": COUNTER[0], "input": COUNTER[1], "output": COUNTER[2], "failed": COUNTER[3]} ARRANGEMENTS = [("code single stage", single(CODE_PROMPT, ALL_CODES), CODE), ("code step back", step_back(CODE_PROMPT, ALL_CODES), CODE), ("degree single stage", single(DEGREE_PROMPT, [(d,) for d in DEGREES]), DEGREE), ("degree step back", step_back(DEGREE_PROMPT, [(d,) for d in DEGREES]), DEGREE)] print(f"{'prompt arrangement':<20}{'accuracy':>8}{'run2':>8}{'diff':>8}{'calls':>7}" f"{'input':>8}{'output':>7}{'failed':>10}") results = {} for name, f, target in ARRANGEMENTS: A, B = measure(f, target), measure(f, target, SEED + 1) results[name] = A print(f"{name:<20}{A['accuracy']:>8.4f}{B['accuracy']:>8.4f}" f"{B['accuracy'] - A['accuracy']:>+8.4f}{A['calls']:>7d}{A['input']:>8d}" f"{A['output']:>7d}{A['failed']:>10d}") for task in ("code", "degree"): a, b = results[task + " single stage"], results[task + " step back"] ta, tb = a["input"] + a["output"], b["input"] + b["output"] print(f"\ncost of the {task} task: calls {a['calls']} -> {b['calls']}, " f"tokens {ta} -> {tb} ({tb / ta:.2f}x)") K = results["code step back"] print(f"principle stage: right principle {K['n_right']}/{len(TICKETS)}, code accuracy under the " f"right principle {K['right_acc']:.4f}, under a wrong one {K['wrong_acc']:.4f} " f"({len(TICKETS) - K['n_right']} tickets)") print("code accuracy when a principle is imposed:", end="") for name in sorted(PRINCIPLES): A = measure(lambda x, t, a=name: (call(PRINCIPLES[a] + " " + CODE_PROMPT + x["text"], ALL_CODES, t)["response"], None), CODE) print(f" {name} {A['accuracy']:.4f}", end="") print() ``` ``` prompt arrangement accuracy run2 diff calls input output failed code single stage 0.5417 0.4792 -0.0625 48 4889 249 1 code step back 0.7500 0.6458 -0.1042 96 8185 339 2 degree single stage 0.7708 0.7500 -0.0208 48 1913 94 1 degree step back 0.7708 0.7917 +0.0208 96 5209 186 2 cost of the code task: calls 48 -> 96, tokens 5138 -> 8524 (1.66x) cost of the degree task: calls 48 -> 96, tokens 2007 -> 5395 (2.69x) principle stage: right principle 45/48, code accuracy under the right principle 0.8000, under a wrong one 0.0000 (3 tickets) code accuracy when a principle is imposed: gozlem 0.7708 kanal 0.0000 sira 0.0625 sure 0.1458 ``` In the code task the two-stage prompt goes **from 0.5417 to 0.7500**, and in the second run from 0.4792 to 0.6458. The gain is **+0.2083** and **+0.1667** in the two runs; the arrangements' own two-run differences are 0.0625 and 0.1042. Since the gain is larger than both, this difference **counts as measured** — not all the techniques in this topic pass this test. The cost is certain too: calls from 48 to **96**, input tokens from 4889 to 8185, total tokens **1.66 times**, the cost per correct code from 198 to 237 tokens. The technique both worked and became more expensive. The last two lines tell the source of the gain and the real risk of the technique. The first stage found the right rule in **45** of 48 tickets, and code accuracy in those 45 is **0.8000**. In all three tickets where a wrong rule was chosen, the code came out wrong — a small number, but the last line proves it alone. When the four rules are imposed in turn, the second stage gives **0.7708** with `gozlem`, 0.1458 with `sure`, 0.0625 with `sira` and **0.0000** with `kanal`; three of them are **below the single-stage 0.5417**. Step-back prompting does not add reasoning to the model, it adds **a new place that can be wrong** to the prompt. The bottom two rows are the limit of the technique. In the degree task the single-stage prompt gives 0.7708 and 0.7500, the step-back prompt 0.7708 and 0.7917; the difference is **0.0000** in one run and +0.0417 in the other, and both are of the order of the arrangements' own run differences. **Counts as unmeasured.** In return the calls double again and the tokens rise **2.69 times** — a worse ratio than in the code task, because as the actual prompt gets shorter the share of the principle text grows. The degree is written word for word in the ticket; the solution does not get stuck at the wrong level of abstraction. **Step-back prompting pays off when the source of the error is the choice of evidence; if the evidence is already chosen correctly, it is only twice the calls.** #### Summary - Step-back prompting splits the call in two: the first call asks for the **general rule** the ticket falls under, the second call writes that rule at its head and asks for the solution. There is no correction in between. - In the code task accuracy goes **from 0.5417 to 0.7500** (in the second run from 0.4792 to 0.6458), and since the gain is larger than the arrangements' two-run differences it counts as measured. - The cost is certain: calls **from 48 to 96**, total tokens **1.66 times**, the cost per correct code from 198 to 237 tokens. - The gain depends on the first stage: the right rule was found in 45 tickets and accuracy there is **0.8000**. When the rules are imposed one by one the second stage varies **between 0.7708 and 0.0000**; all three wrong rules are below the single-stage prompt. - In the task that asks only for the degree the difference stays **between 0.0000 and +0.0417** and cannot be measured, while the tokens rise to **2.69 times**. Raising the level of abstraction has no return for a component that is written word for word. #### Next Step Four lessons built four separate prompt arrangements, and all four tried to win from the same place: **the model taking more steps inside itself**. In all of them the steps were taken entirely inside the simulator; the only thing coming from outside was the text of the ticket, and that text was already at hand in the first call. In this lesson the limit of that appeared. What the right rule was was not written in the ticket; the first stage derived it from the simulator's own tendency and was wrong on three tickets. A situation where the rule could be **read from outside** — looking into a fictional document pool, querying a record — was never covered. A prompt arrangement in which one of the steps is **looking outside** is the topic of the next lesson: asking for thought and action in the same prompt, and in how many responses this produces a valid action. ### Reason and Act Source: https://academia.sh/en/courses/prompt-engineering/reason-and-act Measuring what it takes to ask for the thought field and the action field together in the same prompt: on 120 fictional support tickets, when an action field is requested only 0.1000 of the responses produce a valid action field; when the action set is written into the prompt, 0.5250; when the thought is cut to a single sentence, 0.8000. The cost rises from 5300 to 8060 in input tokens and falls from 645.8 tokens to 102.1 tokens per valid action field. The two-run comparison leaves the difference of writing the action set unmeasurable. The first four lessons of this topic asked for the same thing in four different ways: let the model take more steps **inside itself** before giving the response. The steps became a chain, were duplicated, branched and abstracted; in all four, everything happened inside the model. Yet one of the steps that solve a ticket is not thinking but **looking outside** — searching a document, fetching a past record, opening a new record. This lesson measures **how that step is asked for in the prompt**: when a **thought field** and an **action field** are requested together in the same prompt, how many responses produce a valid action field, and what it costs. What is measured is not the action itself. No action is run in this lesson; the tool scaffolding, the memory and the loop are left to this curriculum's AI Agents course. - **RT41.** The code does not call a real model endpoint; it calls a **language model simulator** written with the standard library. The support tickets are **fictional** too: 120 records, five topics. - **RT42.** Seed **20260218**. The same prompt gives the same response on the same seed; uncertainty is measured by **changing the seed**, and the second run is 20260219. - **RT43.** The action names are fictional and there are three of them; none is run, the response text is read and left there. - **RT44.** Three numbers are counted separately: **label accuracy**, the **share of responses carrying an action field** and the **share of valid action fields**. To count as valid, the name must come from the closed set and the argument must be closed with a parenthesis. - **RT45.** The simulator's format branch depends on the prompt: when an action field is requested, field patterns are produced; when only a thought is requested, thought patterns are produced; all the thought patterns carry a label. - **RT46.** Writing the action names into the prompt changes the pattern weights. It is the action-set counterpart of writing the label set into the prompt. - **RT47.** The largest output is **24 tokens**. When a long thought and the action field do not fit into the same response, the stop condition **cuts the field in the middle** and the action name is corrupted in the truncated response. - **RT48.** The four prompts differ only in **text**. The model, the task, the set and the seed are fixed. #### Two Fields in the Same Prompt The task did not change: each of 120 fictional tickets will be assigned to one of five topics. The only thing that changes is the text of the prompt. **P1** is where this topic left off: think first, then write the label. **P2** adds an action field to the same prompt. **P3** states the action set explicitly. **P4** adds one more sentence to this: keep the thought short. ```python # SIMULATOR. No real model endpoint is called; it is a language model simulator written with the # standard library. The support tickets and the action set are FICTIONAL; no action runs. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUT_TOKENS = 240, 24 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): out = [] for piece in text.lower().replace(",", " , ").split(): while len(piece) > 5: out.append(piece[:5]) piece = piece[5:] out.append(piece) return out def trace_id(text): h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] PATTERNS = { "fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"], } EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim", "acil bilgi almam gerekiyor", "kayit numaram elimde yok"] TICKETS = [] for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] text = PATTERNS[k][int(r() * 3)] if r() < 0.34: text += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": text, "topic": k}) CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CUE = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "borc": "odeme", "odedi": "odeme", "ucret": "odeme", "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet", "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"} TRAP = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} FREE_FORM = ["talebiniz {k} konusuyla ilgili gorunuyor", "bu kayit {k} basligina girer , ilgili ekibe yonlendirildi", "kaydiniz alindi , konuyu inceleyip donus yapacagiz"] COUNTER = [0] ACTIONS = ["belge_ara", "gecmis_getir", "kayit_ac"] # FICTIONAL action names THOUGHTS = ["dusunce : metinde {k} isaretleri var | etiket : {k}", "dusunce : once konu belirlendi sonra etiket yazildi | etiket : {k}", "dusunce : kaydiniz alindi , {k} basligina girer"] FIELDS = ["dusunce : metinde {k} isaretleri var ve gecmis kayit kontrol edilmeli " "| eylem : belge_ara ( {k} )", "dusunce : {k} konusunda once gecmis kayit gerekli | eylem : gecmis_getir ( {k} )", "dusunce : bu kayit {k} basligina girer , dogrudan yanit yazilabilir"] FIELDS_SHORT = ["dusunce : {k} isareti var | eylem : belge_ara ( {k} )", "dusunce : gecmis gerekli | eylem : gecmis_getir ( {k} )", "dusunce : {k} basligina girer"] def call(prompt, seed=SEED, temperature=0.6, max_out=MAX_OUT_TOKENS): COUNTER[0] += 1 toks = tokenize(prompt) if len(toks) > WINDOW: return {"status": "context_overflow", "input": len(toks), "output": 0, "response": "", "truncated": 0} if COUNTER[0] % 37 == 0: return {"status": "rate_limit", "input": len(toks), "output": 0, "response": "", "truncated": 0} r = rng(seed ^ trace_id(prompt)) if len(toks) > 90 and r() < 0.04: return {"status": "timeout", "input": len(toks), "output": 0, "response": "", "truncated": 0} candidates = [k for k in CONCEPTS if k[:5] in toks] if len(candidates) < 3: candidates = CONCEPTS p = {k: 0.15 for k in candidates} for t in toks: for table, w in ((CUE, 1.0), (TRAP, 1.3)): if table.get(t) in p: p[table[t]] += w if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) else: e = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [v / sum(e) for v in e])] if "eylem" in toks: # thought + action field explicit = all(a[:5] in toks for a in ACTIONS) # is the action set written w = [0.46, 0.38, 0.16] if explicit else [0.30, 0.22, 0.48] y = (FIELDS_SHORT if "kisa" in toks else FIELDS)[pick(r(), w)].format(k=k) elif "dusun" in toks: # thought field only y = THOUGHTS[pick(r(), [0.42, 0.26, 0.32])].format(k=k) elif "tek" in toks: y = k else: y = FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k) out_toks, cut = tokenize(y), 0 if len(out_toks) > max_out: # stop condition out_toks, cut = out_toks[:max_out], 1 y = " ".join(out_toks) return {"status": "done", "input": len(toks), "output": len(out_toks), "response": y, "truncated": cut} def read_label(y): for s in y.lower().split(): if s in TOPICS: return s return None def read_action(y): # the action field is read, no action is run if "eylem :" not in y: return None p = y.split("eylem :", 1)[1].split() if len(p) < 3 or p[0] not in ACTIONS or p[1] != "(" or ")" not in p[2:]: return None return p[0] LABEL_PROMPT = "Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife , kesinti . " ACT_PROMPT = "Once dusun , sonra yapilacak eylemi yaz . " ACTION_SET = "Eylem su kumeden secilir: belge_ara , gecmis_getir , kayit_ac . " P1 = LABEL_PROMPT + "Once dusun , sonra etiketi yaz . Talep: " P2 = LABEL_PROMPT + ACT_PROMPT + "Talep: " P3 = LABEL_PROMPT + ACT_PROMPT + ACTION_SET + "Talep: " P4 = LABEL_PROMPT + ACT_PROMPT + ACTION_SET + "Dusunceyi kisa tut . Talep: " t = TICKETS[6] print(f"FICTIONAL ticket {t['id']} ({t['topic']}): {t['text']}") for name, prefix in (("P1 thought only", P1), ("P2 action field", P2), ("P3 set written", P3), ("P4 thought short", P4)): s = call(prefix + t["text"]) print(f"\n{name}: input {s['input']} tok, output {s['output']} tok, " f"truncated {s['truncated']}, label {read_label(s['response'])}, " f"valid action {read_action(s['response'])}") print(f" {s['response']}") ``` ``` FICTIONAL ticket 4007 (sayac): sayac uzerindeki endeks ile fatura ayni degil P1 thought only: input 42 tok, output 14 tok, truncated 0, label sayac, valid action None dusunce : metinde sayac isaretleri var | etiket : sayac P2 action field: input 44 tok, output 16 tok, truncated 0, label sayac, valid action None dusunce : bu kayit sayac basligina girer , dogrudan yanit yazilabilir P3 set written: input 60 tok, output 16 tok, truncated 0, label ariza, valid action None dusunce : bu kayit ariza basligina girer , dogrudan yanit yazilabilir P4 thought short: input 65 tok, output 16 tok, truncated 0, label sayac, valid action gecmis_getir dusunce : gecmis gerekli | eylem : gecmis_getir ( sayac ) ``` On a single ticket four prompts gave four different results. P2 and P3 asked for an action field, and the response returned a sentence of reasoning instead; on top of that, in P3 the label slipped too. Only P4 produced both the right label and a readable action field. A single ticket is not a measurement, but it says where the measurement should look: **the presence of the action field and its validity are two separate numbers.** #### In How Many Responses a Valid Action Field Appears The second block runs the four prompts on all 120 tickets and measures each prompt with two seeds. ```python def measure(prefix, seed=SEED): COUNTER[0] = 0 tin = tout = hits = with_field = valid = cut = 0 for x in TICKETS: s = call(prefix + x["text"], seed=seed) tin, tout, cut = tin + s["input"], tout + s["output"], cut + s["truncated"] hits += read_label(s["response"]) == x["topic"] with_field += "eylem :" in s["response"] valid += read_action(s["response"]) is not None n = len(TICKETS) return {"accuracy": hits / n, "with_field": with_field / n, "valid": valid / n, "truncated": cut / n, "input": tin, "output": tout, "calls": COUNTER[0], "valid_n": valid} NAMES = ["P1 thought", "P2 +action", "P3 +set", "P4 +short"] PREFIXES = [P1, P2, P3, P4] R1 = [measure(p) for p in PREFIXES] R2 = [measure(p, seed=SEED + 1) for p in PREFIXES] print(f"{'measure (seed 20260218)':<24}" + "".join(f"{a:>12}" for a in NAMES)) for name, k, fmt in (("label accuracy", "accuracy", "{:.4f}"), ("carrying action field", "with_field", "{:.4f}"), ("valid action field", "valid", "{:.4f}"), ("truncated response share", "truncated", "{:.4f}"), ("input tokens", "input", "{:d}"), ("output tokens", "output", "{:d}")): print(f"{name:<24}" + "".join(fmt.format(x[k]).rjust(12) for x in R1)) print(f"{'total tokens':<24}" + "".join( str(x["input"] + x["output"]).rjust(12) for x in R1)) print(f"{'per valid action field':<24}" + "".join( ("-" if x["valid_n"] == 0 else f"{(x['input'] + x['output']) / x['valid_n']:.1f}").rjust(12) for x in R1)) print(f"\n{'second run (20260219)':<24}" + "".join(f"{a:>12}" for a in NAMES)) for name, k in (("label accuracy", "accuracy"), ("valid action field", "valid")): print(f"{name:<24}" + "".join(f"{x[k]:.4f}".rjust(12) for x in R2)) largest = max(abs(a["valid"] - b["valid"]) for a, b in zip(R1, R2)) print(f"\nlargest two-run difference (valid action field): {largest:.4f}") for i in range(3): d1 = R1[i + 1]["valid"] - R1[i]["valid"] d2 = R2[i + 1]["valid"] - R2[i]["valid"] verdict = "measured" if min(abs(d1), abs(d2)) > largest else "unmeasured" print(f" {NAMES[i]} -> {NAMES[i + 1]}: diff {d1:+.4f} / {d2:+.4f} {verdict}") ``` ``` measure (seed 20260218) P1 thought P2 +action P3 +set P4 +short label accuracy 0.6750 0.5917 0.4833 0.6750 carrying action field 0.0000 0.3167 0.8833 0.8000 valid action field 0.0000 0.1000 0.5250 0.8000 truncated response share 0.0000 0.2167 0.3583 0.0000 input tokens 5300 5540 7460 8060 output tokens 1729 2210 2584 1745 total tokens 7029 7750 10044 9805 per valid action field - 645.8 159.4 102.1 second run (20260219) P1 thought P2 +action P3 +set P4 +short label accuracy 0.8083 0.4500 0.5917 0.7333 valid action field 0.0000 0.3250 0.3417 0.9333 largest two-run difference (valid action field): 0.2250 P1 thought -> P2 +action: diff +0.1000 / +0.3250 unmeasured P2 +action -> P3 +set: diff +0.4250 / +0.0167 unmeasured P3 +set -> P4 +short: diff +0.2750 / +0.5917 measured ``` The second and third rows are the real distinction of this lesson. In P3, **0.8833** of the responses carry an action field, but only **0.5250** are valid. The **0.3583** in between stands word for word in the row right below: **truncated response share 0.3583.** The source of the loss is not the text of the prompt but the output length limit. The thought and the action field share the 24-token budget of the same response; when the thought runs long, the stop condition falls right on the action field and what is left behind is a remnant such as `eylem : belge _ara (` (act : find_doc ( ), whose format looks right but whose argument is not closed. A field **looking present** and its **being readable** are not the same thing. P4 closes this loss: a single sentence that cuts the thought to one sentence carries the truncated response share **from 0.3583 to 0.0000** and the valid action field **from 0.5250 to 0.8000**. What changed in the text is not even about the action, it is about the length of the thought. #### Cost and the Unmeasurable Difference Input tokens went **from 5300 to 8060**; writing the action set and the keep-it-short sentence at the head of every ticket brought the prompt to **1.52 times**. Output tokens rose to **2584** in P3 and fell to **1745** in P4: the short thought reduced both the generated text and the truncation. The number of calls is 120 in all four prompts, so the whole cost is tokens. The real comparison is in the last row: **from 645.8 tokens to 102.1 tokens per valid action field**. The last block applies the curriculum's measurement rule. The largest difference between two runs in the valid action field is **0.2250**. Writing the action set into the prompt added **+0.4250** in the first run and **+0.0167** in the second; since the two runs do not agree, this difference **counts as unmeasured**. It was known that writing the label set into the prompt works, but that the same move works for the action set as well **could not be shown by this measurement**. Shortening the thought stayed above the band in both runs: **+0.2750 and +0.5917**. In label accuracy no change produced a consistent difference; the two-run difference is at most **0.1417**, and the strongest candidate, the move from P3 to P4, stays right at the edge of the band with **+0.1917 and +0.1416**. **Asking for an action field did not move accuracy; it gained a machine-readable action field, and the cost was paid in tokens.** #### Summary - Asking for thought and action together in the same prompt cannot be measured with a single number: the **share of responses carrying an action field** and the **share of valid action fields** are counted separately. - When the action field was first requested, only **0.1000** of the responses produced a valid field; it rose to 0.5250 when the action set was written and to **0.8000** when the thought was shortened. - The **0.3583** gap between presence and validity in P3 is exactly the **truncated response share**: the two fields share the same 24-token output budget and the limit falls on the field. - Cost: input tokens **5300 → 8060**, **645.8 → 102.1 tokens** per valid action field. The number of calls did not change. - When the two-run rule is applied, the difference of writing the action set stayed **unmeasurable** (+0.4250 and +0.0167); only the difference of shortening the thought was measured. #### Next Step In this lesson and in the previous four, every extra step the model took was **visible**: the thought field was inside the response, its tokens were counted, and what happened when it was cut could be read. The decision to shorten the step could be made only because the step was seen. If a class of models produces these steps inside the call without putting them into the response text at all, what is left: the steps cannot be read and their length cannot be limited, but **their tokens are still paid for**. The next lesson measures this class by its structural difference — two classes on the same task, side by side in accuracy and total tokens — and reports on which task the **invisible token** gained anything and on which the difference stayed unmeasurable. ### Reasoning Models Source: https://academia.sh/en/courses/prompt-engineering/reasoning-models The yardstick for the class of models that produce invisible intermediate steps before writing a response: on the same 120 fictional support tickets, in the single-field task the accuracy difference between the two classes stays at −0.0333 and +0.1167; because the two-run difference is 0.1167 it counts as unmeasured, and tokens rise 1.75 times. In the two-field task the difference rises above the band with +0.1000 and +0.1667; the single-pass class contradicts its own label in the code field in 0.2333 of the responses. The cost per correct response rises from 52.0 to 95.1 and from 83.9 to 110.1 tokens. In the previous lesson every extra step the model took was visible: the thought field was inside the response, its tokens were counted, what happened when it was cut could be read, and the decision to shorten the thought could be made only for that reason. The distinguishing feature of the class measured in this lesson is exactly the absence of this. The class **produces intermediate steps** before writing the response; the text of these steps never returns to the caller, their length cannot be limited by the prompt and the stop condition does not touch them — but they are **paid for as output tokens**. This is what will be measured: **the cost of the invisible token.** This is not a product comparison. No model, version or provider is named; the two classes are defined only by a **structural difference**. The single-pass class reads the prompt and writes the response; the reasoning class inserts steps we do not see. As in self-consistency, the number of calls does not increase — both classes make **one call** per record, and what is paid for there is not visible samples but invisible tokens. - **RT49.** The code does not call a real model endpoint; it calls a **language model simulator** written with the standard library. The support tickets and the document codes are **fictional**. - **RT50.** Seed **20260218**, second run **20260219**. The same prompt gives the same response on the same seed. - **RT51.** The reasoning class produces **one intermediate step per field**. In the single-field task the second step is a **verification pass**: the same candidate set is scored a second time and the response is written from the result of the last step. - **RT52.** The tokens of the intermediate steps are added to the **output tokens** but do not enter the response text. **The stop condition cuts only the visible response**; the invisible steps cannot be limited. - **RT53.** The single-pass class fills the two fields with **two separate draws**: the second field is not tied to the result of the first. In the reasoning class the second intermediate step makes a **mapping** from the result of the first step. - **RT54.** Both tasks use the same 120 fictional tickets. **Task A** asks for one field (the label), **task B** asks for two fields (the label and the document code). - **RT55.** The document codes come from a fictional document pool and each topic has a single code; for a correct response **both fields** must be correct. - **RT56.** The response format is the same in both classes and is imposed. The only measured difference is the class itself. #### The Structural Difference Between the Two Classes The first block builds the simulator and separates the two classes with two functions. `call_standard` reads the prompt and writes the response. `call_reasoning` inserts intermediate steps, accumulates the tokens of the steps in the `hidden` field and writes only the result into the response text. ```python # SIMULATOR. No real model endpoint is called; it is a language model simulator written with the # standard library. The support tickets and the document codes are FICTIONAL. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUT_TOKENS = 240, 24 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): out = [] for piece in text.lower().replace(",", " , ").split(): while len(piece) > 5: out.append(piece[:5]) piece = piece[5:] out.append(piece) return out def trace_id(text): h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] CODES = {"fatura": "FT", "ariza": "AR", "sayac": "SY", "tarife": "TR", "kesinti": "KS"} PATTERNS = { "fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"], } EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim", "acil bilgi almam gerekiyor", "kayit numaram elimde yok"] TICKETS = [] for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] text = PATTERNS[k][int(r() * 3)] if r() < 0.34: text += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": text, "topic": k, "code": CODES[k]}) CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CUE = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "borc": "odeme", "odedi": "odeme", "ucret": "odeme", "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet", "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"} TRAP = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} STEPS = ["ara adim : metinde {k} izleri var , konu buna gore belirlendi", "ara adim : {k} konusunun belge kodu {d} olarak eslendi"] COUNTER = [0] def score(toks, pool): candidates = [k for k in pool if k[:5] in toks] if len(candidates) < 3: candidates = list(pool) p = {k: 0.15 for k in candidates} for t in toks: for table, w in ((CUE, 1.0), (TRAP, 1.3)): if table.get(t) in p: p[table[t]] += w return p def choose(p, r, temperature): if temperature <= 0.0: return max(sorted(p), key=lambda a: p[a]) e = [math.exp(p[a] / temperature) for a in sorted(p)] return sorted(p)[pick(r(), [v / sum(e) for v in e])] def limit_check(toks, r): if len(toks) > WINDOW: return "context_overflow" if COUNTER[0] % 37 == 0: return "rate_limit" if len(toks) > 90 and r() < 0.04: return "timeout" return "done" def reply(status, toks, y, hidden=0): if status != "done": return {"status": status, "input": len(toks), "output": 0, "hidden": 0, "response": ""} out = tokenize(y) if len(out) > MAX_OUT_TOKENS: # the stop condition cuts only the visible part out, y = out[:MAX_OUT_TOKENS], " ".join(out[:MAX_OUT_TOKENS]) return {"status": status, "input": len(toks), "output": len(out) + hidden, "hidden": hidden, "response": y} def body(prompt): # the part of the prompt after the instruction return prompt.split("Talep: ")[-1] def call_standard(prompt, seed=SEED, temperature=0.6): """STANDARD CLASS: single pass, no intermediate step.""" COUNTER[0] += 1 toks = tokenize(prompt) r = rng(seed ^ trace_id(prompt)) status = limit_check(toks, r) if status != "done": return reply(status, toks, "") k = choose(score(toks, CONCEPTS), r, temperature) if "kod" not in toks: return reply(status, toks, f"etiket : {k}") k2 = choose(score(toks, CONCEPTS), r, temperature) # a separate draw for the second field return reply(status, toks, f"etiket : {k} | kod : {CODES.get(k2, 'XX')}") def call_reasoning(prompt, seed=SEED, temperature=0.6): """REASONING CLASS: produces invisible intermediate steps before writing the response. The text of the intermediate steps does not return to the caller but is counted as output tokens.""" COUNTER[0] += 1 toks = tokenize(prompt) r = rng(seed ^ trace_id(prompt)) status = limit_check(toks, r) if status != "done": return reply(status, toks, "") g = tokenize(body(prompt)) k = choose(score(g, TOPICS), r, temperature) # 1st intermediate step hidden = len(tokenize(STEPS[0].format(k=k))) if "kod" not in toks: k = choose(score(g, TOPICS), r, temperature) # 2nd intermediate step: verification hidden += len(tokenize(STEPS[0].format(k=k))) return reply(status, toks, f"etiket : {k}", hidden) code = CODES[k] # 2nd intermediate step: mapping hidden += len(tokenize(STEPS[1].format(k=k, d=code))) return reply(status, toks, f"etiket : {k} | kod : {code}", hidden) def read_fields(y): label = code = None p = y.split() for i, s in enumerate(p): if s == "etiket" and i + 2 < len(p) and p[i + 2] in TOPICS: label = p[i + 2] if s == "kod" and i + 2 < len(p): code = p[i + 2] return label, code A = "Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife , kesinti . Talep: " B = ("Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife , kesinti . " "Ayrica etiketin belge kodunu kod alanina yaz . Talep: ") t = TICKETS[9] print(f"FICTIONAL ticket {t['id']}: {t['text']} (topic {t['topic']}, code {t['code']})") for name, f in (("standard class", call_standard), ("reasoning class", call_reasoning)): for task, prompt in (("A one field", A), ("B two fields", B)): s = f(prompt + t["text"]) print(f"{name:<16}{task:<13} output {s['output']:>3} tok " f"(hidden {s['hidden']:>2}) | {s['response']}") ``` ``` FICTIONAL ticket 4010: hangi tarife basamagindayim ogrenmek istiyorum kayit numaram elimde yok (topic tarife, code TR) standard class A one field output 4 tok (hidden 0) | etiket : sayac standard class B two fields output 9 tok (hidden 0) | etiket : tarife | kod : TR reasoning class A one field output 36 tok (hidden 31) | etiket : tarife reasoning class B two fields output 39 tok (hidden 30) | etiket : tarife | kod : TR ``` Four rows and a single observation on a single record: **the responses have the same format, the outputs are not the same size.** The reasoning class's visible response is five tokens and its output is thirty-six tokens. The thirty-one tokens in between cannot be read, cannot be shortened, cannot be inspected. The previous lesson's lever is not at hand here, because the text to be shortened is not inside the response. #### Two Classes on the Same Task The measurement is made on two tasks. **Task A** asks for one field and the prompt already lists the five labels. **Task B** asks for one more field: the document code of the label. Both use the same 120 fictional tickets and a single call is made for each record. ```python def measure(model_class, prefix, two_fields, seed=SEED): COUNTER[0] = 0 tin = tout = hid = right = agree = 0 for x in TICKETS: s = model_class(prefix + x["text"], seed=seed) tin, tout, hid = tin + s["input"], tout + s["output"], hid + s["hidden"] label, code = read_fields(s["response"]) if two_fields: right += label == x["topic"] and code == x["code"] agree += code is not None and label is not None and code == CODES[label] else: right += label == x["topic"] n = len(TICKETS) return {"accuracy": right / n, "agreement": agree / n if two_fields else None, "input": tin, "visible": tout - hid, "hidden": hid, "total": tin + tout, "right": right} JOBS = [("A standard", call_standard, A, False), ("A reasoning", call_reasoning, A, False), ("B standard", call_standard, B, True), ("B reasoning", call_reasoning, B, True)] R1 = [measure(f, p, two) for _, f, p, two in JOBS] R2 = [measure(f, p, two, seed=SEED + 1) for _, f, p, two in JOBS] print(f"{'measure (seed 20260218)':<24}" + "".join(f"{j[0]:>12}" for j in JOBS)) for name, k, fmt in (("accuracy", "accuracy", "{:.4f}"), ("two-field agreement", "agreement", "{:.4f}"), ("input tokens", "input", "{:d}"), ("visible output tokens", "visible", "{:d}"), ("hidden output tokens", "hidden", "{:d}"), ("total tokens", "total", "{:d}")): print(f"{name:<24}" + "".join(("-" if x[k] is None else fmt.format(x[k])).rjust(12) for x in R1)) print(f"{'per correct response':<24}" + "".join( f"{x['total'] / x['right']:.1f}".rjust(12) for x in R1)) print(f"\n{'second-run accuracy':<24}" + "".join(f"{x['accuracy']:.4f}".rjust(12) for x in R2)) for name, i in (("task A", 0), ("task B", 2)): d1 = R1[i + 1]["accuracy"] - R1[i]["accuracy"] d2 = R2[i + 1]["accuracy"] - R2[i]["accuracy"] band = max(abs(R1[i]["accuracy"] - R2[i]["accuracy"]), abs(R1[i + 1]["accuracy"] - R2[i + 1]["accuracy"])) verdict = "measured" if min(abs(d1), abs(d2)) > band else "unmeasured" ratio = R1[i + 1]["total"] / R1[i]["total"] print(f"{name}: class difference {d1:+.4f} / {d2:+.4f}, two-run difference {band:.4f} " f"-> {verdict} tokens {ratio:.2f} times") ``` ``` measure (seed 20260218) A standard A reasoning B standard B reasoning accuracy 0.7833 0.7500 0.6750 0.7750 two-field agreement - - 0.7667 0.9750 input tokens 4340 4340 5780 5780 visible output tokens 550 547 1017 1017 hidden output tokens 0 3671 0 3438 total tokens 4890 8558 6797 10235 per correct response 52.0 95.1 83.9 110.1 second-run accuracy 0.6667 0.7833 0.6333 0.8000 task A: class difference -0.0333 / +0.1167, two-run difference 0.1167 -> unmeasured tokens 1.75 times task B: class difference +0.1000 / +0.1667, two-run difference 0.0417 -> measured tokens 1.51 times ``` **In task A the difference could not be measured.** The class difference is **−0.0333** in the first run and **+0.1167** in the second; its sign is not even constant and its size does not rise above the two-run difference (**0.1167**). The reason can be read from the table: the prompt already lists the five labels, so the job of narrowing the candidate set has been done **in the prompt text**. The intermediate step does the same narrowing one more time and adds no new information. In return, the invisible output tokens go **from 0 to 3671** and the total reaches **1.75 times**. This is the plainest form of the curriculum's second claim: the technique multiplied the tokens and did not move the number. **In task B the situation reverses.** The class difference is positive and large in both runs — **+0.1000** and **+0.1667** — while the two-run difference is **0.0417**. The reason sits in the `two-field agreement` row: only **0.7667** of the single-pass class's responses have a code field **consistent with its own label**. In other words, the model often finds the label right and writes the code from another topic; the two fields are not tied to each other. In the reasoning class this ratio is **0.9750** — the remaining share is the three calls that hit the rate limit and produced no response at all. Producing the second field from the **result** of the first closes the whole loss. #### The Cost of the Invisible Token The last row is the bill for this lesson. **From 52.0 tokens to 95.1 per correct response** (A) and **from 83.9 to 110.1** (B). In task B the reasoning class really did raise accuracy and **was still more expensive per correct response**, because while accuracy rose 1.15 times, the total tokens rose 1.51 times. The number of calls is 120 in both classes; the whole increase is in **output tokens**, and most of those tokens were never read. The shape of the cost matters too. The visible output tokens are almost the same in the two classes — 550 and 547 in task A. The difference is entirely in the invisible part: **3671 and 3438 tokens.** This ties the prompt designer's hands: a length constraint can be written but does not work on the invisible step, a format can be imposed but the invisible step has no format, truncated responses can be counted but the invisible step is not truncated. The only lever left is **the task itself**: is there a decision the intermediate step will narrow, or has the prompt already done that narrowing. **The invisible intermediate step gains something only when it does a job the prompt could not do; if the prompt has already done the job, only tokens are left.** #### Summary - The reasoning model class is defined not by a product but by a **structural difference**: it produces intermediate steps before the response, the steps do not return to the caller and are **paid for as output tokens**. - The number of calls does not increase. In self-consistency the cost was in the calls and the samples could be seen; here the cost is inside a single call and it **cannot be seen**. - In the single-field task the difference between the two classes is **−0.0333 and +0.1167**, and the two-run difference is **0.1167**: the difference **counts as unmeasured**, yet tokens rose **1.75 times**. - In the two-field task the difference is **+0.1000 and +0.1667**, above the band (**0.0417**). Its source is `two-field agreement`: the single-pass class gets the code field right with its own label in only **0.7667** of the responses, the reasoning class in **0.9750**. - The cost per correct response is **52.0 → 95.1** and **83.9 → 110.1 tokens**. Even in the task where it gained, the unit cost rose. #### Next Step Over six lessons a single prompt was worked over: its steps were chained, duplicated, branched, abstracted, an action field was put into it and finally it was handed over to a class whose steps are not visible. All of them shared one assumption — there is **a single text** looking at the question, and the good result will come from making that text even longer. Having **several separate, independent prompts** look at the same question, each producing its own response, and combining these was never tried. The next lesson measures this and asks the real question: do different prompts **go wrong on the same examples**? If the errors overlap, combining gains nothing, and this can be counted before combining. ### Prompt Ensemble Source: https://academia.sh/en/courses/prompt-engineering/prompt-ensemble The same 120 fictional support tickets are examined with three separate instructions, and the assumption the ensemble rests on is tested: do different prompts make different errors. The three prompts alone give 0.7750, 0.6583 and 0.7833, on nine records all three are wrong at once and the ceiling comes out at 0.9250. The majority vote stays at 0.8000: 0.1250 below the ceiling, only 0.0167 above the best single prompt, and this gain is below the 0.0333 to 0.1333 difference the same prompts show between two runs. The cost is written in number of calls: 120 calls become 360 calls, and 61.1 tokens per correct label become 180.7 tokens. The previous lesson measured how far a single prompt can be stretched: a model class that produces invisible intermediate steps before the response spent more tokens on the same task, and it was written that the difference stayed unmeasurable in one task. What the six lessons have in common is this — there was always **a single prompt**. That prompt was lengthened, forced into step-by-step solving, branched, split in two. Having **several separate prompts** look at the same question was never tried. This is the scaffolding of this lesson: the same task, the same simulator, the same set; three separately written instructions and, at the end, a vote. Its name is the **prompt ensemble**. The procedure of the majority vote was measured in the ensemble methods lesson of the Supervised Learning course and is not repeated. Two things are added: writing the cost **in number of calls** and testing the assumption the ensemble rests on. That assumption is this — different prompts make **different errors**. The number of this lesson is how far the assumption holds: on how many records all three prompts are wrong **at once**. - **RT61.** The task, the set and the simulator are fixed. The only thing that changes is the **text of the instruction**; all three instructions carry the same five labels and the same format constraint. - **RT62.** The code does not call a real model endpoint; it calls a **simulator** written with the standard library. The support tickets and the topics are **fictional**. - **RT63.** The three instructions carry three separate design decisions: the plain label set, stating a role and asking for the operation area first. - **RT64.** The criterion is **label accuracy**. If a label cannot be read from the response, the record counts as wrong. - **RT65.** The ensemble's decision is the **majority vote**. If all three labels differ, the label of the first prompt holds; the number of these cases is printed separately. - **RT66.** **Overlap** is this: if all three prompts are wrong on a record at once, that record is one the ensemble cannot reach. The share of the remaining records is the ensemble's **ceiling**. - **RT67.** The cost is written **in number of calls**: a three-member ensemble spends three calls for each record. The unit cost per token and per correct label is put next to it. - **RT68.** Every measure is printed in **two runs** (seeds 20260218 and 20260219). If the ensemble's gain is smaller than the two-run difference it counts as **unmeasured**. #### Three Prompts Looking at the Same Task ```python # SIMULATOR -- FICTIONAL. This code does not call a real model endpoint; it calls a language model # simulator written with the standard library. The tickets are fictional too. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUT_TOKENS = 240, 24 # context window and largest output, in tokens def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): # FICTIONAL tokenizer: pieces of at most five characters out = [] for piece in text.lower().replace(",", " , ").split(): while len(piece) > 5: out.append(piece[:5]) piece = piece[5:] out.append(piece) return out def trace_id(text): # a trace tied to the content of the prompt, independent of the run h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] PATTERNS = { "fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"], } EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim", "acil bilgi almam gerekiyor", "kayit numaram elimde yok"] TICKETS = [] for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] text = PATTERNS[k][int(r() * 3)] if r() < 0.34: text += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": text, "topic": k}) CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CUE = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "borc": "odeme", "odedi": "odeme", "ucret": "odeme", "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet", "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"} TRAP = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} # misleading links FREE_FORM = ["talebiniz {k} konusuyla ilgili gorunuyor", "bu kayit {k} basligina girer , ilgili ekibe yonlendirildi", "kaydiniz alindi , konuyu inceleyip donus yapacagiz"] COUNTER = [0] def call(prompt, seed=SEED, temperature=0.6, max_out=MAX_OUT_TOKENS): COUNTER[0] += 1 toks = tokenize(prompt) if len(toks) > WINDOW: return {"status": "context_overflow", "input": len(toks), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: return {"status": "rate_limit", "input": len(toks), "output": 0, "response": ""} r = rng(seed ^ trace_id(prompt)) if len(toks) > 90 and r() < 0.04: return {"status": "timeout", "input": len(toks), "output": 0, "response": ""} candidates = [k for k in CONCEPTS if k[:5] in toks] # concepts named in the prompt if len(candidates) < 3: candidates = CONCEPTS p = {k: 0.15 for k in candidates} for t in toks: for table, w in ((CUE, 1.0), (TRAP, 1.3)): if table.get(t) in p: p[table[t]] += w if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) # most likely choice else: e = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [v / sum(e) for v in e])] y = k if "tek" in toks else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k) out = tokenize(y) if len(out) > max_out: # stop condition out = out[:max_out] y = " ".join(out) return {"status": "done", "input": len(toks), "output": len(out), "response": y} def read_label(y): # extracting the label from the response for s in y.lower().split(): if s in TOPICS: return s return None THREE = { "plain": ("Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife , " "kesinti . Yanit olarak tek etiket yaz . Talep: "), "role": ("Sen destek masasi gorevlisisin . Talebi fatura , ariza , sayac , tarife , " "kesinti etiketlerinden birine ata , tek etiket yaz . Talep: "), "area": ("Once talebin hangi islem alanina girdigini belirle , sonra fatura , " "ariza , sayac , tarife , kesinti icinden tek etiket yaz . Talep: "), } def run_prompt(instruction, seed=SEED): COUNTER[0] = 0 labels, tin, tout, failed = [], 0, 0, 0 for x in TICKETS: s = call(instruction + x["text"], seed=seed) tin, tout, failed = tin + s["input"], tout + s["output"], failed + (s["status"] != "done") labels.append(read_label(s["response"])) return {"labels": labels, "input": tin, "output": tout, "calls": COUNTER[0], "failed": failed} def accuracy(labels): return sum(e == x["topic"] for e, x in zip(labels, TICKETS)) / len(TICKETS) RUN1 = {name: run_prompt(y) for name, y in THREE.items()} RUN2 = {name: run_prompt(y, SEED + 1) for name, y in THREE.items()} print(f"{'prompt':<7} {'tokens':>8} {'run 1':>8} {'run 2':>8} {'diff':>8} " f"{'input':>7} {'output':>6}") for name in THREE: a, b = accuracy(RUN1[name]["labels"]), accuracy(RUN2[name]["labels"]) print(f"{name:<7} {len(tokenize(THREE[name])):>8d} {a:>8.4f} {b:>8.4f} {b - a:>+8.4f} " f"{RUN1[name]['input']:>7d} {RUN1[name]['output']:>6d}") i = next(i for i in range(len(TICKETS)) if len({RUN1[name]["labels"][i] for name in THREE}) == 3) print(f"\nticket {TICKETS[i]['id']} ({TICKETS[i]['topic']}): {TICKETS[i]['text']}") print("labels of the three prompts: " + " , ".join(f"{name} -> {RUN1[name]['labels'][i]}" for name in THREE)) ``` ``` prompt tokens run 1 run 2 diff input output plain 32 0.7750 0.6917 -0.0833 5300 192 role 37 0.6583 0.7917 +0.1333 5900 210 area 34 0.7833 0.7500 -0.0333 5540 201 ticket 4023 (fatura): bu ayki fatura tutari gecen aya gore cok yuksek geldi bu konuda memnun degilim labels of the three prompts: plain -> fatura , role -> tarife , area -> sayac ``` Three prompts gave three separate numbers; the largest difference between them is **0.1250**. The real information is in the second run column. In the first run the best prompt is `area` (0.7833), the worst is `role` (0.6583); in the second run the ranking reverses and `role` takes the lead with **0.7917**. The difference of the same prompt between two runs is **0.0833**, **0.1333** and **0.0333**: the difference between prompts and the wobble of the same prompt within itself are **of the same size**. At this scale "the best prompt" is not a stable concept. The last two rows are the hope of the ensemble. Ticket 4023 is on the topic bill, and the three prompts give three separate labels; one of them knew it right. The ensemble's whole claim is this observation — if the errors fall in different places, vote combining eliminates them. #### Overlap of Errors and the Ceiling ```python from collections import Counter def vote(row): # majority; if all three labels differ, the first prompt decides s = Counter(e for e in row if e is not None) if not s: return None top = max(s.values()) return next((e for e in row if e is not None and s[e] == top), None) def analyze(RUN): rows = [[RUN[name]["labels"][i] for name in THREE] for i in range(len(TICKETS))] hit = [[row[j] == TICKETS[i]["topic"] for j in range(3)] for i, row in enumerate(rows)] ens = [vote(row) for row in rows] return {"ens": ens, "all_wrong": sum(not any(x) for x in hit), "ceiling": sum(any(x) for x in hit) / len(TICKETS), "no_consensus": sum(len(set(row)) == 3 for row in rows), "wrong_consensus": sum(1 for i, row in enumerate(rows) if Counter(row).most_common(1)[0][1] >= 2 and ens[i] != TICKETS[i]["topic"]), "accuracy": accuracy(ens), "calls": sum(RUN[n]["calls"] for n in THREE), "tokens": sum(RUN[n]["input"] + RUN[n]["output"] for n in THREE)} C1, C2 = analyze(RUN1), analyze(RUN2) best = max(accuracy(RUN1[n]["labels"]) for n in THREE) print(f"{'measure':<28} {'run 1':>9} {'run 2':>9}") for name, k, fmt in (("all three wrong", "all_wrong", "{:d}"), ("at least one right (ceiling)", "ceiling", "{:.4f}"), ("three different labels", "no_consensus", "{:d}"), ("agreement on a wrong label", "wrong_consensus", "{:d}"), ("ensemble accuracy", "accuracy", "{:.4f}")): print(f"{name:<28} {fmt.format(C1[k]):>9} {fmt.format(C2[k]):>9}") print(f"\nbest single prompt {best:.4f} -> ensemble {C1['accuracy']:.4f} " f"({C1['accuracy'] - best:+.4f}); distance to ceiling " f"{C1['accuracy'] - C1['ceiling']:+.4f}") single = RUN1["area"]["input"] + RUN1["area"]["output"] print(f"cost: calls {RUN1['area']['calls']} -> {C1['calls']} " f"({C1['calls'] / RUN1['area']['calls']:.1f} times), tokens {single} -> " f"{C1['tokens']} ({C1['tokens'] / single:.2f} times)") print(f"per correct label {single / (best * 120):.1f} -> " f"{C1['tokens'] / (C1['accuracy'] * 120):.1f} tokens") print("\nlabel changes between runs (over 120 records):") for name in THREE: print(f" {name:<9} {sum(a != b for a, b in zip(RUN1[name]['labels'], RUN2[name]['labels'])):>3d}") print(f" {'ensemble':<9} {sum(a != b for a, b in zip(C1['ens'], C2['ens'])):>3d}") ``` ``` measure run 1 run 2 all three wrong 9 16 at least one right (ceiling) 0.9250 0.8667 three different labels 1 8 agreement on a wrong label 24 18 ensemble accuracy 0.8000 0.8000 best single prompt 0.7833 -> ensemble 0.8000 (+0.0167); distance to ceiling -0.1250 cost: calls 120 -> 360 (3.0 times), tokens 5741 -> 17343 (3.02 times) per correct label 61.1 -> 180.7 tokens label changes between runs (over 120 records): plain 39 role 43 area 40 ensemble 24 ``` The first row answers this lesson's question directly. On **9** of the 120 records all three prompts are wrong at once; the overlap is **0.0750**. So the highest number the ensemble can reach is **0.9250**, and a gap of **0.1417** stands above the best single prompt. Overlap is low, the ceiling is high: up to here the ensemble's rationale stands. The realized number is **0.8000**, **0.1250** below the ceiling. The gap sits in a place that can be counted: the **agreement on a wrong label** row says the majority met on a wrong label in 24 records. Nine of these are records nobody could know anyway; on the remaining **15 records one prompt knew the right answer and lost the vote**. In a three-member ensemble the single member that knows is always in the minority. Overlap determines the ceiling, the vote determines how much of that ceiling will be reached; the two are separate numbers, and a report that writes only the first overstates the ensemble's gain by two times. The second run column shows how much the measure itself wobbles: only the seed changed, the overlap rose from 9 to **16**, the ceiling fell to **0.8667**. Overlap is a quantity that cannot be read from a single run. #### What Three Times the Calls Bought The gain is **+0.0167** against the best single prompt. The difference of the same prompts between two runs was between 0.0333 and 0.1333. **The gain is only half of even the smallest of the run differences; by this curriculum's rule it counts as unmeasured.** In the second run the gain is **+0.0083**; what its sign will be in the next run is not known. The cost, on the other hand, is measured and certain. The number of calls rose from 120 to **360** — exactly **three times**, because the number of members is directly the multiplier. The total tokens rose **3.02 times**, and the crumb in accuracy does not close this multiplier: the cost per correct label goes **from 61.1 tokens to 180.7 tokens**. A scaffolding that is three times as expensive giving a gain that cannot be measured is the exact counterpart of this curriculum's second claim — **reasoning techniques are expensive and do not pay off on every task.** The last table shows the one thing actually received in return for three calls. The single prompts change the labels of 39, 43 and 40 records between two runs; the ensemble changes them on **24 records**. What the vote does is not raise accuracy but **stabilize the output**, and it leaves that half done. The accuracy of the two runs came out **0.8000** and **0.8000**; under these two equal numbers there are 24 labels that changed. Assuming it is stable because the totals are equal is the very mistake this curriculum warns against. A prompt ensemble pays off when the members' errors really fall in different places. Here all three instructions carried the same five labels and the same format constraint; their differences stayed in the wording, not in the decision. Saying the same information in three separate sentences does not produce three viewpoints, it produces three bills. #### Summary - Three separate instructions give **0.7750**, **0.6583** and **0.7833** on the same task; in the second run the ranking reverses. The difference between prompts and the run difference are of the same size. - Overlap is counted: on **9** of the 120 records all three are wrong at once, the ceiling is **0.9250**. In the second run these numbers are **16** and **0.8667**. - The majority vote stays at **0.8000** — **0.1250** below the ceiling. On **15 records** one prompt knew the right answer and lost the vote; the majority met on a wrong label in 24 records. - The gain is **+0.0167** against the best single prompt, and the two-run difference is between 0.0333 and 0.1333. The gain counts as unmeasured. - The cost is written in number of calls: 120 calls **360 calls**, 61.1 tokens per correct label **180.7 tokens**. The only thing received was the change between runs falling from 40 to **24**. #### Next Step In this lesson three prompts were written by hand and which one was better changed with the run. There was a criterion, a set and candidate prompts at hand; only the choice was made by eye and by looking at a single run. Turning the same job into a loop is possible: candidates are produced, all are scored with the same criterion and the highest is chosen. The next lesson builds this loop and asks a single question — as the search budget grows, how much of the number that rises on the validation set carries over to a test set that has never been seen. ### Automatic Prompt Optimization Source: https://academia.sh/en/courses/prompt-engineering/automatic-prompt-optimization Instead of being written by hand, the prompt is searched with a criterion: ten candidates are produced mechanically from four parts, and the fictional support tickets are split into a 40-record validation set and an 80-record test set. As the search budget goes from one candidate to ten, the best score on validation rises from 0.7750 to 0.9000 and the test score of the same candidate falls from 0.6750 to 0.6500: validation gain +0.1250, test gain -0.0250, and the chosen candidate's optimism margin 0.2500. The search goes from 40 calls to 400 calls, the candidate that is truly best on the test (0.8000) is never chosen, and in the second run the search chooses a different candidate. In the previous lesson three prompts were written by hand, all three were measured on the same set and the best was chosen by eye. The basis of that choice was a single run, and in the second run the ranking reversed. The hand-done part of this job is actually narrow: producing the candidates is a combination job, scoring is a loop, choosing is a comparison. All three can be written. When they are written, what comes out is **automatic prompt optimization** — searching for the prompt with a criterion. The search itself is not the subject of this lesson. The procedure of choosing the best over a set of candidates and **the optimism of the best score on the validation set as the tuning budget grows** were measured in the hyperparameter search lesson of the Supervised Learning course and are not repeated. What is searched here is not a tuning value but a **text**, and what gives the score is a nondeterministic component. The lesson's single question is this: as the search budget grows, does the number gained on validation carry over to a test set that has never been seen. - **RT69.** The code does not call a real model endpoint; it calls a **simulator** written with the standard library. The support tickets and the topics are **fictional**. - **RT70.** **The number of candidates does not exceed 10.** The candidates are produced mechanically from four parts: the role sentence, the prefix sentence, the order of the label list and the format sentence. - **RT71.** The set is split in two: **validation 40 records**, **test 80 records**. The test set is never seen during the search and no choice is made by looking at it. - **RT72.** The criterion is **label accuracy**; if a label cannot be read from the response, the record counts as wrong. - **RT73.** Search rule: when the budget is $b$, the one with the highest validation among the **first $b$ candidates** is chosen. In a tie, the one that comes first in the list wins. - **RT74.** The cost of the search is written **in number of calls**. Every candidate is run on the whole validation set; for budget $b$ the cost is $b \times 40$ calls. - **RT75.** The **optimism margin** is the difference between the chosen candidate's validation score and its test score. - **RT76.** Two runs are printed. With seed 20260219 the search is repeated from scratch; whether the chosen candidate changed and how much the test score of the same candidate wobbled is written. #### Ten Candidates, Two Sets ```python # SIMULATOR -- FICTIONAL. This code does not call a real model endpoint; it calls a language model # simulator written with the standard library. The tickets are fictional too. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUT_TOKENS = 240, 24 # context window and largest output, in tokens def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): # FICTIONAL tokenizer: pieces of at most five characters out = [] for piece in text.lower().replace(",", " , ").split(): while len(piece) > 5: out.append(piece[:5]) piece = piece[5:] out.append(piece) return out def trace_id(text): # a trace tied to the content of the prompt, independent of the run h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] PATTERNS = { "fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"], } EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim", "acil bilgi almam gerekiyor", "kayit numaram elimde yok"] TICKETS = [] for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] text = PATTERNS[k][int(r() * 3)] if r() < 0.34: text += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": text, "topic": k}) CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CUE = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "borc": "odeme", "odedi": "odeme", "ucret": "odeme", "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet", "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"} TRAP = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} # misleading links FREE_FORM = ["talebiniz {k} konusuyla ilgili gorunuyor", "bu kayit {k} basligina girer , ilgili ekibe yonlendirildi", "kaydiniz alindi , konuyu inceleyip donus yapacagiz"] COUNTER = [0] def call(prompt, seed=SEED, temperature=0.6, max_out=MAX_OUT_TOKENS): COUNTER[0] += 1 toks = tokenize(prompt) if len(toks) > WINDOW: return {"status": "context_overflow", "input": len(toks), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: return {"status": "rate_limit", "input": len(toks), "output": 0, "response": ""} r = rng(seed ^ trace_id(prompt)) if len(toks) > 90 and r() < 0.04: return {"status": "timeout", "input": len(toks), "output": 0, "response": ""} candidates = [k for k in CONCEPTS if k[:5] in toks] # concepts named in the prompt if len(candidates) < 3: candidates = CONCEPTS p = {k: 0.15 for k in candidates} for t in toks: for table, w in ((CUE, 1.0), (TRAP, 1.3)): if table.get(t) in p: p[table[t]] += w if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) # most likely choice else: e = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [v / sum(e) for v in e])] y = k if "tek" in toks else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k) out = tokenize(y) if len(out) > max_out: # stop condition out = out[:max_out] y = " ".join(out) return {"status": "done", "input": len(toks), "output": len(out), "response": y} def read_label(y): # extracting the label from the response for s in y.lower().split(): if s in TOPICS: return s return None ROLES = ["", "Sen destek masasi gorevlisisin . ", "Kayit gorevlisi olarak yanitla . "] PREFIXES = ["", "Once talebin hangi islem alanina girdigini belirle , sonra ", "Ornek: endeks tutmuyor -> sayac . "] LISTS = ["fatura , ariza , sayac , tarife , kesinti", "kesinti , tarife , sayac , ariza , fatura", "ariza , kesinti , fatura , sayac , tarife"] FORMATS = ["Yanit olarak tek etiket yaz .", "Tek etiket yaz , aciklama ekleme .", "Cevabin tek sozcuk olsun , listedeki etiketlerden biri ."] CANDIDATES, PARTS = [], [] r = rng(SEED + 777) while len(CANDIDATES) < 10: # candidates are produced mechanically, the seed is fixed d = [int(r() * 3) for _ in range(4)] y = (ROLES[d[0]] + PREFIXES[d[1]] + "Talebi su etiketlerden birine ata: " + LISTS[d[2]] + " . " + FORMATS[d[3]] + " Talep: ") if y not in CANDIDATES: CANDIDATES.append(y) PARTS.append("".join(str(x) for x in d)) VAL, TEST = TICKETS[:40], TICKETS[40:] def score_prompt(instruction, subset, seed=SEED): right = 0 for x in subset: right += read_label(call(instruction + x["text"], seed=seed)["response"]) == x["topic"] return right / len(subset) COUNTER[0] = 0 VAL_SCORES = [score_prompt(y, VAL) for y in CANDIDATES] SEARCH_CALLS = COUNTER[0] TEST_SCORES = [score_prompt(y, TEST) for y in CANDIDATES] print(f"{'candidate':>9} {'parts':>6} {'tokens':>8} {'validation':>10} {'test':>8} " f"{'diff':>8}") for i, y in enumerate(CANDIDATES): print(f"{i + 1:>9d} {PARTS[i]:>6} {len(tokenize(y)):>8d} {VAL_SCORES[i]:>10.4f} " f"{TEST_SCORES[i]:>8.4f} {TEST_SCORES[i] - VAL_SCORES[i]:>+8.4f}") print(f"\nvalidation set {len(VAL)} records, test set {len(TEST)} records") print(f"search calls {SEARCH_CALLS}, validation mean {sum(VAL_SCORES) / 10:.4f}, " f"test mean {sum(TEST_SCORES) / 10:.4f}") ``` ``` candidate parts tokens validation test diff 1 2110 53 0.7750 0.6750 -0.1000 2 2201 51 0.7250 0.7250 +0.0000 3 1210 50 0.7500 0.6500 -0.1000 4 1000 41 0.6500 0.8000 +0.1500 5 0212 47 0.4250 0.4625 +0.0375 6 1101 56 0.9000 0.6500 -0.2500 7 0121 47 0.6000 0.6750 +0.0750 8 0202 47 0.5250 0.6000 +0.0750 9 1020 41 0.7750 0.7375 -0.0375 10 0100 45 0.7750 0.6875 -0.0875 validation set 40 records, test set 80 records search calls 400, validation mean 0.6900, test mean 0.6663 ``` The last line closes a wrong explanation from the start. The means of the two sets are **0.6900** and **0.6663**; there is no systematic gulf between them. That is, the test set is not harder and its distribution is not different. Even so, the `diff` column goes both ways at once and reaches a size of **0.2500**. The split is not between the sets, it is **per candidate**. The sixth candidate is first on validation by a wide margin with **0.9000**, and third from last on the test with **0.6500**. The fourth candidate is the exact opposite: low on validation with 0.6500, first on the test with **0.8000**. In a set of forty records a single record is worth 0.0250; the **0.1250** difference that separates the sixth candidate from the first one is nothing more than where five records fell. The `parts` column gives the four digits of the candidate: the role sentence, the prefix sentence, the order of the label list, the format sentence. The fourth candidate is `1000`, the ninth candidate is `1020`; they differ only in the third digit. The same role sentence, the same absence of a prefix, the same format sentence, the same five labels — the only difference is the **order the labels are written in** and the test scores are **0.8000** and **0.7375**. The score wobbling by 0.0625 with the information the prompt carries unchanged already opens the question of what this search measures. #### As the Search Budget Grows ```python def select_best(vs): # budget b: the one with the highest validation among the first b return [max(range(b), key=lambda i: vs[i]) for b in range(1, len(vs) + 1)] CHOSEN = select_best(VAL_SCORES) print(f"{'budget':>6} {'chosen':>6} {'validation':>10} {'test':>8} {'search calls':>12}") for b, i in enumerate(CHOSEN, 1): print(f"{b:>6d} {i + 1:>6d} {VAL_SCORES[i]:>10.4f} {TEST_SCORES[i]:>8.4f} {b * len(VAL):>12d}") first, last = CHOSEN[0], CHOSEN[-1] print(f"\nbudget 1 -> 10: validation {VAL_SCORES[last] - VAL_SCORES[first]:+.4f}, " f"test {TEST_SCORES[last] - TEST_SCORES[first]:+.4f}") print(f"optimism margin of the chosen candidate {VAL_SCORES[last] - TEST_SCORES[last]:+.4f}") best_test = max(range(10), key=lambda i: TEST_SCORES[i]) print(f"best candidate on test {best_test + 1} ({TEST_SCORES[best_test]:.4f}); its validation " f"{VAL_SCORES[best_test]:.4f}, the search never chose it") COUNTER[0] = 0 VAL_SCORES2 = [score_prompt(y, VAL, SEED + 1) for y in CANDIDATES] TEST_SCORES2 = [score_prompt(y, TEST, SEED + 1) for y in CANDIDATES] last2 = select_best(VAL_SCORES2)[-1] print(f"\nsecond run (seed {SEED + 1}): chosen candidate {last2 + 1}, validation " f"{VAL_SCORES2[last2]:.4f}, test {TEST_SCORES2[last2]:.4f}") print(f"test of candidate {last + 1} over two runs {TEST_SCORES[last]:.4f} -> " f"{TEST_SCORES2[last]:.4f} ({TEST_SCORES2[last] - TEST_SCORES[last]:+.4f})") print(f"is the test ranking of the ten candidates the same in both runs: " f"{sorted(range(10), key=lambda i: -TEST_SCORES[i]) == sorted(range(10), key=lambda i: -TEST_SCORES2[i])}") ``` ``` budget chosen validation test search calls 1 1 0.7750 0.6750 40 2 1 0.7750 0.6750 80 3 1 0.7750 0.6750 120 4 1 0.7750 0.6750 160 5 1 0.7750 0.6750 200 6 6 0.9000 0.6500 240 7 6 0.9000 0.6500 280 8 6 0.9000 0.6500 320 9 6 0.9000 0.6500 360 10 6 0.9000 0.6500 400 budget 1 -> 10: validation +0.1250, test -0.0250 optimism margin of the chosen candidate +0.2500 best candidate on test 4 (0.8000); its validation 0.6500, the search never chose it second run (seed 20260219): chosen candidate 9, validation 0.7500, test 0.7875 test of candidate 6 over two runs 0.6500 -> 0.8500 (+0.2000) is the test ranking of the ten candidates the same in both runs: False ``` Two columns of the table move in opposite directions. As the budget goes from one to ten, the validation column rises **from 0.7750 to 0.9000** — the search works, does its job, finds a higher number with every new candidate. On the same rows the test column falls **from 0.6750 to 0.6500**. The validation gain is **+0.1250**, the test gain is **−0.0250**. Raising the search budget ten times gained nothing on data that has never been seen; it brought a small loss. The cost is in the same table and beyond dispute: **from 40 calls to 400 calls**. An extra 360 calls were spent and the only thing obtained in return was **a 0.1250 rise in the number to be reported**. The chosen candidate's **optimism margin is 0.2500**: the prompt that reads 0.9000 on validation gives 0.6500 on the records never seen. Delivering this prompt as "the 0.9000 prompt" means making up a quarter of the measure. The third row shows the blindness of the search. The candidate that is truly best on the test is the fourth (**0.8000**) and the search did not choose it under any budget; because it scored 0.6500 on validation, it was eliminated in the very first comparison. The search did not find the good prompt, it found the prompt that **fits the validation set best**. #### Two Runs and What the Search Measures The last three rows probe the ground of the whole exercise. When the seed is changed and the search is run from scratch, the chosen candidate slides **from the sixth to the ninth**: the same candidates, the same sets, the same rule, a different winner. The ninth candidate differed from the fourth, which was first on the test, only by the label order. Moreover, the sixth candidate's test score wobbles **from 0.6500 to 0.8500**, that is by **+0.2000**. The validation difference the search won over ten candidates was 0.1250; **the wobble of a single candidate from run to run is above that**. The last row completes this: the test ranking of the ten candidates is not the same in the two runs. The conclusion drawn from this does not forbid the search, it corrects its yardstick. The best score on the validation set is not the quality of a prompt, it is **the residue of a search budget**; it grows optimistic as the budget grows, and this optimism is not visible unless it is measured with a separate number on a set that has never been touched. The same holds for the second run of a nondeterministic component. The curriculum's rule takes its final form here: **a prompt changed without measurement is not an improvement but a guess** — and a prompt measured on a single set in a single run is the same. #### Summary - Ten candidates are produced mechanically from four parts; the validation set is 40, the test set 80 records. The means of the two sets are **0.6900** and **0.6663**, so the split is not between the sets but per candidate. - As the search budget goes from one to ten, validation rises **from 0.7750 to 0.9000** and the test of the same candidate falls **from 0.6750 to 0.6500**: **−0.0250** in return for **+0.1250**. - The cost is written in number of calls: the search goes **from 40 calls to 400 calls**. - The chosen candidate's **optimism margin is 0.2500**. The candidate that is truly best on the test (0.8000) is not chosen under any budget because it scored 0.6500 on validation. - In the second run the search chooses a different candidate and the sixth candidate's test wobbles by **+0.2000** — the run difference of a single candidate is larger than the difference the search won over ten candidates. #### Next Step Over sixteen lessons what to say to the prompt was arranged and the model was asked how to think: instruction, role, format, example, negative constraint, step-by-step solving, multiple samples, branching, stepping back, action field, ensemble and now search. Under all of them sat something silent — the response coming from the model was **free text**. In every lesson the label was extracted from inside the response with a reading procedure; a response that could not be read was counted wrong and that share was counted separately. Every improvement made on the prompt side amounted to **asking for** the format of the output. The next topic moves to this side: the settings the response is generated with, where the response stops, imposing the format, validating the incoming output against the schema and the business rule, and how much of the response that gets caught by validation is rescued by retrying. ## Output Control Measuring how the output of an uncertain generator is constrained in form and in content: the combined behavior of temperature, top-p (nucleus) and top-k, and which of them narrows the difference between two runs; the repetition rate and the number of prematurely cut responses when a repetition penalty and a stop sequence are added; the schema match rate and token cost of free text, a format description and an imposed schema; the schema and business rule violations that validation catches, the responses that retrying recovers and the extra calls it costs; a wrong response that is sometimes fixed by a better instruction and sometimes only by missing information; how much balancing through the prompt narrows the subgroup gap and that it does not solve data bias. ### Sampling Parameters Source: https://academia.sh/en/courses/prompt-engineering/sampling-parameters Measuring temperature, top-p (nucleus) and top-k together on the same prompt: label accuracy 0.6417 and two-run difference 0.5667 on the base setting, 0.8750 and 0.0417 at temperature 0.2, 0.8417 and 0.1333 at top-p 0.50, 0.7833 and 0.3583 at top-k 3, 0.8667 and 0.0083 when all three narrow together. The fixed cut leaves three candidates on average while the top-p cut leaves 1.20 candidates. Input tokens stay constant at 5300 and the call count stops at 120, so the cost of the change is thirteen output tokens. The previous sixteen lessons treated the text of the prompt as a design object: the instruction was arranged, examples were placed, constraints were written, the model was asked to reason step by step. In all of these lessons what was measured was **what was written into the prompt**. The response itself always came back as free text, and how it was produced was never asked. This topic opens from there. A response is drawn from a distribution, and that draw has three settings: the distribution temperature, top-p (nucleus) and top-k. These three settings were defined in the Introduction to AI Engineering course; their definitions are not repeated here. The question of this lesson is different: **what happens when all three are used at the same time**, which one helps on which task and which one narrows the **difference between two runs**. - **OC1.** The code is a **simulator** and does not call a real model endpoint. The support tickets, the topics and the document vocabulary are **fictional**; seed **20260218**. - **OC2.** The previous prompt is fixed: an instruction that lists the label set and asks for a single label. This prompt **never changes** during the whole lesson; the only thing that changes is the sampling setting of the call. - **OC3.** The criterion is twofold. In the labeling task **label accuracy** (higher is better) is measured, in the free-form response task the **number of distinct responses** (diversity). - **OC4.** The **two-run difference** is the **share of records whose decision changes** between seed 20260218 and seed 20260219 on the same 120 tickets. A difference that a change adds counts as unmeasured if it is smaller than this number. - **OC5.** Temperature reshapes the distribution; it **does not shorten the candidate list**. Top-p (nucleus) and top-k, on the other hand, **cut the list**. - **OC6.** The number of candidates is printed as well: the number of candidates left by the fixed cut and the number left by the adaptive cut appear in the same table. - **OC7.** The simulator's internal cue table has **misleading ties**; no sampling setting corrects them. - **OC8.** The cost is written in **tokens**: input tokens, output tokens and the number of calls. #### The Separate Jobs of the Three Settings The three settings work in the same place: after the scores are turned into a probability distribution and right before the draw. Temperature is a divisor and sharpens or flattens the distribution. Top-k cuts the list **at a fixed number** from the start. Top-p (nucleus) looks not at a number but at the **sum of probabilities** and takes candidates until it reaches the threshold. ```python # SIMULATOR -- FICTIONAL. No real model endpoint is called; the code below is a language # model simulator written with the standard library. The tickets are FICTIONAL too. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUT = 240, 24 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): # FICTIONAL tokenizer, a piece has at most five characters out = [] for piece in text.lower().replace(",", " , ").split(): while len(piece) > 5: out.append(piece[:5]) piece = piece[5:] out.append(piece) return out def trace_id(text): # depends on the prompt, independent of the run h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"), ("faturami odedim ama sistemde borc gorunuyor", "fatura"), ("fatura ayrintisini nereden gorebilirim", "fatura"), ("evde su akmiyor bir ariza var galiba", "ariza"), ("baglanti bozuk oldu ve hic calismiyor", "ariza"), ("ariza kaydi actim ama ekip gelmedi", "ariza"), ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"), ("sayac degisimi icin randevu istiyorum", "sayac"), ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"), ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"), ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"), ("birim ucret abone tipine gore degisiyor mu", "tarife"), ("mahallede kesinti var ne zaman gelecek", "kesinti"), ("planli kesinti duyurusu nereden takip edilir", "kesinti"), ("aksamdan beri kesinti yasiyoruz", "kesinti")] EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim", "acil bilgi almam gerekiyor", "kayit numaram elimde yok"] TICKETS = [] for i in range(120): # 120 FICTIONAL support tickets r = rng(SEED + i) topic = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] text, _ = [x for x in PATTERNS if x[1] == topic][int(r() * 3)] TICKETS.append({"id": 4001 + i, "topic": topic, "text": text + (" " + EXTRAS[int(r() * 4)] if r() < 0.34 else "")}) CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "borc": "odeme", "odedi": "odeme", "ucret": "odeme", "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet", "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"} TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} # misleading cues FREE_FORM = ["talebiniz {k} konusuyla ilgili gorunuyor", "bu kayit {k} basligina girer , ilgili ekibe yonlendirildi", "kaydiniz alindi , konuyu inceleyip donus yapacagiz"] COUNTER = [0] def distribution(toks, temperature, top_p, top_k): """The three settings work here and only here: temperature reshapes the distribution, top-k leaves a fixed number of candidates, top-p (nucleus) looks at the probability sum.""" cands = [k for k in CONCEPTS if k[:5] in toks] p = {k: 0.15 for k in (cands if len(cands) >= 3 else CONCEPTS)} for t in toks: for table, w in ((CUES, 1.0), (TRAPS, 1.3)): if table.get(t) in p: p[table[t]] += w names = sorted(p, key=lambda a: (-p[a], a)) if temperature <= 0.0: return [names[0]], [1.0] # most probable choice e = [math.exp(p[a] / temperature) for a in names] probs = [v / sum(e) for v in e] names, probs = names[:top_k], probs[:top_k] # top-k selection n, mass = 0, 0.0 while mass < top_p and n < len(probs): # top-p (nucleus) sampling mass += probs[n] n += 1 names, probs = names[:n], probs[:n] return names, [v / sum(probs) for v in probs] def call(prompt, seed=SEED, temperature=1.0, top_p=1.0, top_k=9, max_out=MAX_OUT): COUNTER[0] += 1 toks = tokenize(prompt) if len(toks) > WINDOW: return {"status": "context_overflow", "input": len(toks), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: return {"status": "rate_limit", "input": len(toks), "output": 0, "response": ""} r = rng(seed ^ trace_id(prompt)) names, probs = distribution(toks, temperature, top_p, top_k) k = names[pick(r(), probs)] y = k if "tek" in toks else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k) out = tokenize(y) if len(out) > max_out: # stop condition out, y = out[:max_out], " ".join(out[:max_out]) return {"status": "ok", "input": len(toks), "output": len(out), "response": y} def read_label(y): for s in y.lower().split(): if s in TOPICS: return s return None LABEL_PROMPT = ("Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife , " "kesinti . Yanit olarak tek etiket yaz . Talep: ") b = tokenize(LABEL_PROMPT + TICKETS[6]["text"]) print(f"FICTIONAL ticket set {len(TICKETS)} records; sample ticket {TICKETS[6]['id']}") print(f"prompt {len(b)} tokens, actual topic {TICKETS[6]['topic']}") print("setting candidates top three probabilities") for name, t, p, k in (("base 1.0 / 1.00 / 9", 1.0, 1.0, 9), ("temperature 0.2", 0.2, 1.0, 9), ("top-p 0.50", 1.0, 0.5, 9), ("top-k 3", 1.0, 1.0, 3)): a, o = distribution(b, t, p, k) print(f"{name:<24} {len(a):>10} {' '.join(f'{v:.3f}' for v in o[:3])}") ``` ``` FICTIONAL ticket set 120 records; sample ticket 4007 prompt 42 tokens, actual topic sayac setting candidates top three probabilities base 1.0 / 1.00 / 9 5 0.564 0.207 0.076 temperature 0.2 5 0.993 0.007 0.000 top-p 0.50 1 1.000 top-k 3 3 0.665 0.245 0.090 ``` The four rows show three separate jobs. At temperature 0.2 the number of candidates **stays at five** but the highest probability goes from 0.564 to 0.993: the list did not shorten, the distribution sharpened. Top-p 0.50 returns with a single candidate, because the probability of the first candidate is already 0.564 and exceeds the threshold on its own. Top-k 3 brings the list down to three, but when the probabilities of the remaining three candidates are rescaled, the second candidate **rises** from 0.207 to 0.245. Cutting strengthens what remains. The distinction that comes out of this carries the rest of the lesson: temperature is a **reshaping**, top-p (nucleus) is an **adaptive cut**, top-k is a **fixed cut**. #### The Three Settings Together Nine combinations of settings are measured on the same prompt, the same 120 tickets and the same simulator. In the labeling task the accuracy is printed, in the free-form response task the number of distinct responses. The two-run difference is the share of records whose decision changes in the second run, taken with the seed changed. ```python FREE_PROMPT = "Su destek talebine kisa bir yanit yaz . Talep: " SETTINGS = [("base 1.0 / 1.00 / 9", 1.0, 1.0, 9), ("temperature 0.6", 0.6, 1.0, 9), ("temperature 0.2", 0.2, 1.0, 9), ("most probable choice", 0.0, 1.0, 9), ("top-p 0.50", 1.0, 0.5, 9), ("top-p 0.20", 1.0, 0.2, 9), ("top-k 3", 1.0, 1.0, 3), ("top-k 1", 1.0, 1.0, 1), ("all three together", 0.6, 0.5, 3)] def run(instruction, t, p, k, seed): COUNTER[0] = 0 return [read_label(call(instruction + x["text"], seed=seed, temperature=t, top_p=p, top_k=k)["response"]) for x in TICKETS] print("setting accuracy A accuracy B two-run difference distinct responses candidates") for name, t, p, k in SETTINGS: a, b = run(LABEL_PROMPT, t, p, k, SEED), run(LABEL_PROMPT, t, p, k, SEED + 1) diff = sum(1 for x, y in zip(a, b) if x != y) / len(a) acc_a = sum(x == tk["topic"] for x, tk in zip(a, TICKETS)) / len(a) acc_b = sum(x == tk["topic"] for x, tk in zip(b, TICKETS)) / len(b) COUNTER[0] = 0 distinct = len({call(FREE_PROMPT + x["text"], temperature=t, top_p=p, top_k=k)["response"] for x in TICKETS}) cands = sum(len(distribution(tokenize(LABEL_PROMPT + tk["text"]), t, p, k)[0]) for tk in TICKETS) / len(TICKETS) print(f"{name:<22} {acc_a:>10.4f} {acc_b:>10.4f} {diff:>18.4f} {distinct:>18} {cands:>10.2f}") for name, t, p, k in (SETTINGS[0], SETTINGS[-1]): COUNTER[0] = 0 y = [call(LABEL_PROMPT + x["text"], temperature=t, top_p=p, top_k=k) for x in TICKETS] print(f"cost {name:<22} input {sum(z['input'] for z in y)} " f"output {sum(z['output'] for z in y)} calls {COUNTER[0]}") ``` ``` setting accuracy A accuracy B two-run difference distinct responses candidates base 1.0 / 1.00 / 9 0.6417 0.6000 0.5667 15 5.03 temperature 0.6 0.8167 0.8250 0.2833 17 5.03 temperature 0.2 0.8750 0.8917 0.0417 10 5.03 most probable choice 0.8667 0.8667 0.0000 10 1.00 top-p 0.50 0.8417 0.9667 0.1333 15 1.20 top-p 0.20 0.8667 0.8667 0.0000 10 1.00 top-k 3 0.7833 0.7833 0.3583 15 3.00 top-k 1 0.8667 0.8667 0.0000 10 1.00 all three together 0.8667 0.8667 0.0083 10 1.01 cost base 1.0 / 1.00 / 9 input 5300 output 191 calls 120 cost all three together input 5300 output 204 calls 120 ``` The base row is an unmeasurable row. The accuracy looks like **0.6417**, but the two-run difference is **0.5667**: in more than half of the records the decision changes when the seed changes. No prompt improvement measured on this base can be defended, because the difference sought stays below the noise. All three settings narrow this difference, but not to the same extent. Temperature 0.2 brings the difference down to **0.0417** and raises the accuracy to **0.8750**, the highest in the table. What stands out is that this is higher than the most probable choice (**0.8667**) too: sharpening the distribution gave a better result than cutting it off entirely. Top-p 0.50 brings the difference down to **0.1333** while keeping the mean number of candidates at **1.20**; that is, it drops to a single candidate on most tickets and leaves the second candidate open on the few tickets where the evidence is weak. This is the **adaptive** cut. The top-k 3 row is the most instructive row in the table. The number of candidates is fixed at **3.00**, and the two-run difference of **0.3583** is the largest value apart from the base. A fixed cut leaves two rival candidates unnecessarily on the tickets where the evidence is strong, and temperature 1.0 keeps making them compete. Cutting alone does not close the uncertainty; **what the cut is made by** decides. The free-form response column shows the opposite direction. Narrowing gains in labeling and loses here: the number of distinct responses is **17** at temperature 0.6 and falls to **10** in the most narrowed settings; it stays at **15** with top-p 0.50 and top-k 3. The same three settings work in two directions on two tasks. In a task with a single correct answer, uncertainty is a defect; in a task where a separate sentence is expected for each subscriber, uncertainty is **the product itself**. Ten distinct responses remain because the settings do not affect at all the draw that selects the **format** of the response: a setting does not close every source of uncertainty. #### A Change Whose Cost Is Zero Tokens The cost line of this lesson differs from the rest of the course. The previous prompt and the changed prompt are **the exact same text**: the input tokens are **5300** in both settings, the number of calls is **120** in both. The output tokens go from 191 to 204, because the narrowed setting chooses the longer-written labels slightly more often. The total cost is **13 tokens**, that is, 0.11 tokens per request. What is bought in return is measurability itself. On the base setting the two-run difference is 0.5667, while when all three are narrowed together it falls to **0.0083**. In the lessons from here on, a gain of 0.05 added by a prompt change is a meaningful number; on the base setting the same gain could not have been measured. **The rule of the course turns into a precondition here: a prompt changed without being measured is not an improvement but a guess, and measurement is possible only when the two-run difference is small enough.** This also has a limit that must be written down. In the setting where all three are narrowed, the accuracy stays at **0.8667**, and most of the 16 remaining wrong records come from the simulator's misleading ties. Sampling settings close the uncertainty; they **do not close systematic error**. A response that is stably wrong is stable too. #### Summary - Temperature shapes the distribution and does not shorten the candidate list; top-p (nucleus) makes an **adaptive** cut by looking at the probability sum; top-k leaves a **fixed** number of candidates. - On the base setting the label accuracy is 0.6417 but the two-run difference is 0.5667; on this base no prompt change can be measured. - Temperature 0.2 brings the difference down to 0.0417 while raising the accuracy to 0.8750, the highest, and passes the 0.8667 of the most probable choice; sharpening the distribution gave a better result than cutting it off entirely. - Top-k 3 fixes the number of candidates at 3.00 and leaves the two-run difference at 0.3583; top-p 0.50 brings the difference down to 0.1333 with 1.20 candidates. - The same settings lower the diversity in the free-form response task from 17 to 10: narrowing is a gain in one task and a loss in the other. - The cost of the change is 13 output tokens and zero extra calls; the input tokens stay constant at 5300. #### Next Step The sampling settings controlled which concept would be chosen, but never controlled **where the response would end**. In this lesson the output length was cut by a single hard limit, and how many responses were cut was not counted. In the same way, a response repeating the same sentence within itself was never measured either: sharpening the distribution makes the most probable phrase even more probable, and in long responses this turns into self-repetition. The next lesson adds two new controls and measures both with numbers: how much the repetition penalty lowers the repetition rate, how many responses the stop sequence cuts and how many of them it cuts **prematurely**. ### Penalties and Stop Sequences Source: https://academia.sh/en/courses/prompt-engineering/penalties-and-stop-sequences Measuring the repetition penalty and the stop sequence on one response set: on the base scaffolding the repetition rate is 0.3431 and 107 of 120 responses are truncated at the length limit; a negative constraint written into the prompt lowers the rate to 0.2167 while raising the input tokens from 2734 to 3574, and a 0.45 repetition penalty does the same job with zero extra tokens, down to 0.1972. The stop sequence lowers the number of truncated responses to 55, and to 24 together with the penalty, and brings the output tokens from 4073 to 2653, but it cuts twenty-five responses prematurely, before the label sentence is written, and lowers usability from 0.8833 to 0.7417. The two-run difference on the stop sequence row is 0.0543, and the 0.0746 gain over the base is marginal. Last lesson the sampling settings controlled which concept would be chosen. **Where the response would end** was left to a single hard length limit, and how many responses were truncated at that limit was never counted. In the same way, a long response repeating the same sentence within itself was not measured either. This lesson adds two controls and compares both with numbers. The **repetition penalty** lowers the score of a piece that has been produced before. The **stop sequence** ends the generation when a given piece of text is produced. Both are call settings and neither adds a token to the prompt; so the lesson also measures a third path: asking for the same thing **by writing it into the prompt**. - **OC9.** The code is a **simulator**; no real endpoint is called. The tickets and the response sentences are **fictional**, seed **20260218**. - **OC10.** The simulator produces the response **sentence by sentence**. At each step one of six fictional sentences is drawn according to its score; if the draw is unconstrained, the same sentence can come up more than once. - **OC11. Repetition rate** is the share of repeated sentences in a response: one minus the ratio of the number of distinct sentences to the total number of sentences, averaged over 120 responses. - **OC12. Truncated response** is a response that hits the output token limit (34 tokens) and has its last sentence left half-written. **Prematurely cut response** is a response that ends **without the label sentence being written** because of the stop sequence; the two are counted separately. - **OC13.** The label sentence is the sentence that states the topic was recorded. A last sentence left half-written **does not count as complete**; the usability of a response is measured by looking at this sentence. - **OC14.** The negative constraint in the prompt corresponds to a **mild penalty** in the simulator. This is a fictional assumption showing that an instruction is not directly translated into a setting. - **OC15.** Two runs are taken with seeds 20260218 and 20260219; the repetition rate is printed separately for the two runs and the difference between them is the measurability threshold. - **OC16.** The cost is written as input tokens, output tokens and the number of calls; the number of calls never changes in this lesson. #### A Response Repeating Itself A response is not a single choice; it is many choices one after another. At each step the highest-scored piece is a little more probable, and having been chosen once does not make it less probable. The result is a loop that shows itself in long responses: the same sentence keeps coming back until the limit truncates the response. ```python # SIMULATOR -- FICTIONAL. No real model endpoint is called. This block is a language model # simulator that produces the response sentence by sentence; the tickets are FICTIONAL too. import math SEED, M32 = 20260218, 0xFFFFFFFF MAX_OUT, MAX_SENTENCES = 34, 6 # output token limit and sentence limit def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): # FICTIONAL tokenizer, a piece has at most five characters out = [] for piece in text.lower().replace(",", " , ").split(): while len(piece) > 5: out.append(piece[:5]) piece = piece[5:] out.append(piece) return out def trace_id(text): h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"), ("faturami odedim ama sistemde borc gorunuyor", "fatura"), ("fatura ayrintisini nereden gorebilirim", "fatura"), ("evde su akmiyor bir ariza var galiba", "ariza"), ("baglanti bozuk oldu ve hic calismiyor", "ariza"), ("ariza kaydi actim ama ekip gelmedi", "ariza"), ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"), ("sayac degisimi icin randevu istiyorum", "sayac"), ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"), ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"), ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"), ("birim ucret abone tipine gore degisiyor mu", "tarife"), ("mahallede kesinti var ne zaman gelecek", "kesinti"), ("planli kesinti duyurusu nereden takip edilir", "kesinti"), ("aksamdan beri kesinti yasiyoruz", "kesinti")] TICKETS = [] for i in range(120): # 120 FICTIONAL support tickets r = rng(SEED + i) topic = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] TICKETS.append({"id": 4001 + i, "topic": topic, "text": [x for x in PATTERNS if x[1] == topic][int(r() * 3)][0]}) CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti"} SENTENCE_BANK = [("talebiniz alindi", 1.00), ("konu {k} olarak kaydedildi", 1.25), ("ilgili ekibe yonlendirildi", 1.05), ("en kisa surede donus yapilacak", 0.95), ("ek bilgi icin kayit numaranizi hazir bulundurun", 0.80), ("anlayisiniz icin tesekkur ederiz", 0.85)] COUNTER = [0] def respond(prompt, seed=SEED, penalty=0.0, stop=None, max_out=MAX_OUT): """The response is produced sentence by sentence. The penalty lowers the score of a sentence that was used; if the stop sequence appears in a produced sentence, the generation ends without writing that sentence.""" COUNTER[0] += 1 toks = tokenize(prompt) p = {k: 0.15 for k in TOPICS} for t in toks: if CUES.get(t) in p: p[CUES[t]] += 1.0 k = max(sorted(p), key=lambda a: p[a]) if "tekra" in toks: # negative constraint in the prompt penalty = max(penalty, 0.45) r = rng(seed ^ trace_id(prompt)) used, sentences, emitted, truncated, early = {}, [], [], False, False while len(sentences) < MAX_SENTENCES: s = [b - penalty * used.get(c, 0) for c, b in SENTENCE_BANK] e = [math.exp(v / 0.35) for v in s] c = SENTENCE_BANK[pick(r(), [v / sum(e) for v in e])][0].format(k=k) if stop and stop in c: # stop sequence early = not any("konu" in x for x in sentences) break used[c.replace(k, "{k}")] = used.get(c.replace(k, "{k}"), 0) + 1 new = tokenize(c) if len(emitted) + len(new) > max_out: # length limit: truncation emitted, truncated = (emitted + new)[:max_out], True sentences.append(c) break emitted += new sentences.append(c) complete = sentences[:-1] if truncated else sentences return {"sentences": sentences, "tokens": len(emitted), "truncated": truncated, "early": early, "label": any(c.startswith("konu") for c in complete)} INSTRUCTION = "Su destek talebine kisa bir yanit yaz . Talep " SAMPLE = TICKETS[4] o = respond(INSTRUCTION + f"{SAMPLE['id']} : {SAMPLE['text']}") print(f"FICTIONAL ticket set {len(TICKETS)} records; sample ticket {SAMPLE['id']}") print(f"output limit {MAX_OUT} tokens, sentence limit {MAX_SENTENCES}") for c in o["sentences"]: print(f" - {c}") print(f"sentences {len(o['sentences'])}, distinct sentences {len(set(o['sentences']))}, " f"tokens {o['tokens']}, truncated {o['truncated']}, label {o['label']}") d = respond(INSTRUCTION + f"{SAMPLE['id']} : {SAMPLE['text']}", penalty=0.90, stop="ek bilgi") print("same ticket with penalty 0.90 and stop sequence 'ek bilgi':") for c in d["sentences"]: print(f" - {c}") print(f"sentences {len(d['sentences'])}, distinct sentences {len(set(d['sentences']))}, " f"tokens {d['tokens']}, truncated {d['truncated']}, early {d['early']}") ``` ``` FICTIONAL ticket set 120 records; sample ticket 4005 output limit 34 tokens, sentence limit 6 - anlayisiniz icin tesekkur ederiz - ilgili ekibe yonlendirildi - en kisa surede donus yapilacak - talebiniz alindi - ilgili ekibe yonlendirildi - en kisa surede donus yapilacak sentences 6, distinct sentences 4, tokens 34, truncated True, label False same ticket with penalty 0.90 and stop sequence 'ek bilgi': - anlayisiniz icin tesekkur ederiz - ilgili ekibe yonlendirildi - en kisa surede donus yapilacak - talebiniz alindi - konu sayac olarak kaydedildi sentences 5, distinct sentences 5, tokens 31, truncated False, early False ``` The same ticket received two different responses under two settings, and the difference is not a matter of style. The upper response carries two repetitions, with four of six sentences distinct, spends all thirty-four tokens, is truncated at the limit and **never manages to write the label sentence**. The response is unusable: the assigned topic is not in it. The lower response ends at thirty-one tokens with five distinct sentences, and the label sentence makes it to the end. No truncation, no repetition. The causality matters here. Repetition does not only make the response ugly; it **uses up the limited budget and leaves the actual content out**. When repetition combines with the length limit, it stops being a formatting problem and becomes a **loss of content**. #### Counting the Three Paths in the Same Table The same 120 tickets are run under six scaffoldings. The second row is a negative constraint **written into the prompt**; the third and fourth rows are the repetition penalty, which is a **call setting**; the fifth row is the stop sequence, and the sixth is the combination of the two. ```python NEGATIVE_CONSTRAINT = "Ayni cumleyi tekrar etme . " SCAFFOLDING = [("base", INSTRUCTION, 0.0, None), ("negative constraint in prompt", NEGATIVE_CONSTRAINT + INSTRUCTION, 0.0, None), ("repetition penalty 0.45", INSTRUCTION, 0.45, None), ("repetition penalty 0.90", INSTRUCTION, 0.90, None), ("stop sequence", INSTRUCTION, 0.0, "ek bilgi"), ("penalty + stop sequence", INSTRUCTION, 0.90, "ek bilgi")] def measure(instruction, penalty, stop, seed=SEED): COUNTER[0] = 0 rep = trunc = early = usable = n_in = n_out = 0 for x in TICKETS: prompt = instruction + f"{x['id']} : {x['text']}" o = respond(prompt, seed=seed, penalty=penalty, stop=stop) n = len(o["sentences"]) rep += 0.0 if n == 0 else 1 - len(set(o["sentences"])) / n trunc += o["truncated"] early += o["early"] usable += o["label"] n_in += len(tokenize(prompt)) n_out += o["tokens"] return (rep / len(TICKETS), trunc, early, usable / len(TICKETS), n_in, n_out) print("scaffolding repetition A repetition B truncated early has label input output") for name, instruction, pen, stop in SCAFFOLDING: a = measure(instruction, pen, stop) b = measure(instruction, pen, stop, SEED + 1) print(f"{name:<30} {a[0]:>12.4f} {b[0]:>12.4f} {a[1]:>4}/{b[1]:<4} {a[2]:>5}" f" {a[3]:>9.4f} {a[4]:>5} {a[5]:>6}") ``` ``` scaffolding repetition A repetition B truncated early has label input output base 0.3431 0.3750 107/106 0 0.8833 2734 4073 negative constraint in prompt 0.2167 0.1939 112/117 0 0.9083 3574 4072 repetition penalty 0.45 0.1972 0.1944 117/117 0 0.9667 2734 4078 repetition penalty 0.90 0.0892 0.0925 120/120 0 0.9833 2734 4080 stop sequence 0.2685 0.3228 55/59 25 0.7417 2734 2942 penalty + stop sequence 0.0578 0.0715 24/25 20 0.8333 2734 2653 ``` The second and third rows are the real comparison of this lesson. Both ask for the same thing: do not repeat yourself. The negative constraint lowers the repetition rate **from 0.3431 to 0.2167** and raises the input tokens **from 2734 to 3574** — seven tokens per ticket, **840** in total. The repetition penalty lowers the same rate to **0.1972** and the input tokens **stay constant at 2734**. The less effective path is expensive, the more effective path is free. That a target can be written into the prompt does not mean that writing it there is right. When the penalty is raised from 0.45 to 0.90, the repetition falls to **0.0892** and the gain is **0.1080**. On this row the two-run difference is only **0.0033**, so the gain is more than thirty times the difference: an improvement that is comfortably measurable. The same reading gives the opposite result on the fifth row. The stop sequence lowers the repetition from 0.3431 to **0.2685**, a gain of **0.0746**; but the two-run difference of that row is **0.0543**. The gain stays only a little above the difference; a single run would make this row look stronger than it is. **A prompt changed without being measured is not an improvement but a guess**, and measurement requires reading the two runs together. That the two-run difference changes from row to row is also information. On the penalty rows it stays between 0.0033 and 0.0028, because the penalty prevents a second choice of the same sentence to the same extent in every run. On the stop sequence row the difference goes up to 0.0543, because where the response ends depends on a single draw and that draw shifts with the seed. While a scaffolding improves the number it measures, it **can lower the reliability of that number**; a table that does not print two runs does not show this. The table also shows that while the repetition rate falls, **the number of truncated responses rises**: 107 on the base, 120 with penalty 0.90. The penalty reduces repetition but does not shorten the response; on the contrary, the distinct sentences that come in place of the repetition are longer and use up the budget faster. The only scaffolding that solves the length problem is the stop sequence: the truncated responses fall to **55**, and to **24** together with the penalty, and the output tokens drop **from 4073 to 2653**. The total gain is **1420 tokens**, 11.8 per response. #### What the Stop Sequence Cuts Early The cost of the stop sequence is paid not in the output but in the **content**. In the fifth row, twenty-five responses end without the label sentence being written at all: the sentence carrying the stop sequence fell early in the order and the generation stopped there. The share of usable responses falls from the base's **0.8833** to **0.7417**. That is, the 1131 tokens gained from the output tokens were paid for with seventeen usable responses. The last row shows how the two scaffoldings correct each other. When the repetition penalty is added, the early cuts go from 25 to **20** and the usability rises to **0.8333**. The reason is the order: the penalty lowers the score of sentences that were used before, so the label sentence moves toward the beginning of the response and in most responses it has already been written when the stop sequence arrives. Even so, the last row is **still below the base**: 0.8333 against 0.8833. Together the two scaffoldings made the output a third cheaper and lowered the usability. This shows that choosing the stop sequence is a design decision. The sequence is not arbitrary: a piece of text expected to appear **after the mandatory content** in the response must be chosen. The piece chosen here was the beginning of an additional-information sentence expected near the end of the response, and in the twenty-five responses where the expectation did not hold, content was lost. What needs to be measured is not the sequence itself but **how often it appears before the mandatory content**; without printing this number the stop sequence does not count as measured. To wrap up: the base prompt is the plain response instruction: 2734 input tokens, repetition 0.3431, truncated 107, output 4073, usability 0.8833. The changed prompt can be built in two separate ways. Writing a negative constraint into the prompt lowers the repetition to 0.2167 and adds **840 tokens**. Changing the call setting lowers the repetition to 0.0578, the truncated responses to 24 and the output to 2653; **the input tokens and the number of calls never change**, in return twenty responses are cut early and the usability falls below the base. The cost of a scaffolding is not always paid in tokens; sometimes it is paid directly in **usability**. #### Summary - Because the response is produced sentence by sentence, repetition arises by itself: on the base scaffolding the repetition rate is 0.3431 and 107 of 120 responses are truncated at the length limit. - Repetition is not a style problem; it uses up the limited output budget, leaves the label sentence out and makes the response unusable. - A negative constraint written into the prompt lowers the repetition to 0.2167 and adds 840 input tokens; the 0.45 repetition penalty, which is a call setting, lowers the same rate to 0.1972 and adds zero tokens. - When the penalty goes from 0.45 to 0.90 the gain is 0.1080 and the two-run difference is 0.0033; the stop sequence's gain over the base is 0.0746 and the two-run difference is 0.0543, that is, the second row is marginal. - The stop sequence lowers the truncated responses from 107 to 55, and to 24 together with the penalty, and brings the output tokens from 4073 to 2653; its cost is twenty-five prematurely cut responses and usability falling from 0.8833 to 0.7417. #### Next Step These two controls arranged **how long** the response lasted, not **what it looked like**. Truncation decreased, repetition decreased, but the output is still free text: in which sentence of the response the label appears, in which word it is written and what else is next to it is different every time. An output that will be written into a record system cannot be read in this form. The next lesson compares the format of the output on three levels — free text, a format description written into the prompt and an imposed schema — and at each level prints the **rate of responses that match the schema** together with the token cost. ### Structured Output Source: https://academia.sh/en/courses/prompt-engineering/structured-output Measuring free text, a format description written into the prompt and an imposed schema side by side on the same 120 fictional tickets: the rate of schema-conforming responses is 0.0000 in free text, 0.8000 with the format description, 0.8833 when an example is added and 1.0000 with the imposed schema. The two-run difference of the format description row is 0.0833 and, since the gain from adding an example is also 0.0833, that gain counts as unmeasured, and it adds 2040 tokens. Topic hit rate stays fixed at 0.8833 at all four levels. Input tokens rise from 2254 to 5614 with the format description and to 7654 with the example, while the imposed schema stays at 2974; that is, the cheapest prompt gives the highest conformance. The previous two lessons arranged **which concept** the response chose and **where it ended**. The response itself was still a sentence: in which word the label appeared, which phrase stood next to it and where a record field could be read from was different every time. If the output of a support assistant goes not to a person but to a **record system**, a sentence is unusable. What is needed is a record in which certain fields are filled with certain values. This lesson asks for that record on three levels — free text, a **format description** written into the prompt and an **imposed schema** that constrains the generation — and prints the **rate of schema-conforming responses** of the three side by side with the token cost. - **OC17.** The code is a **simulator**; no real endpoint is called. The tickets, the fields and the value sets are **fictional**, seed **20260218**. - **OC18. The schema is four fields:** a ticket number that must be a number, and a topic, a priority and an action, three of which come from closed value sets. The ticket number is written into the prompt, that is, the model does not have to make it up. - **OC19.** The schema validator is **written inside the lesson**; no schema library is used from outside. Four error classes are counted: could not be parsed, missing key, extra key, invalid value. - **OC20. A format description is a tendency, not a guarantee.** When the simulator sees a format description in the prompt, it produces a share of the responses in line with the description; the remaining share falls into one of five broken formats. The fifth **passes the schema**: the fields are in place but the ticket number is a number that is not on record. - **OC21. An imposed schema is a constraint.** In this mode the generation **cannot** leave the schema; the fields are complete and the values come from the closed sets. - **OC22.** Every decision in the simulator comes from **a single separately salted draw**; the decisions are not tied to each other. The topic choice is read only from the **ticket text**, so only the format changes between the format levels. - **OC23. A recordable correct record** is a response that both conforms to the schema and has the correct topic; this is the product criterion of this lesson. - **OC24.** The cost is written as input and output tokens; the number of calls is one per ticket at all three levels. #### Schema, Parser and Validator Schema validation was measured in the Web API Design course and in the Cybersecurity curriculum; the procedure is not repeated here. The only thing added is the identity of the producer: the record being validated comes not from a client but from an **uncertain component**. For this reason the validator is not a security gate but a **measuring tool**: it counts how many responses fall into which class. ```python # SIMULATOR -- FICTIONAL. No real model endpoint is called. The schema validation is also # written inside the lesson; no schema library is used from outside. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): # FICTIONAL tokenizer, a piece has at most five characters out = [] for piece in text.lower().replace(",", " , ").split(): while len(piece) > 5: out.append(piece[:5]) piece = piece[5:] out.append(piece) return out def trace_id(text): h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] PRIORITIES, ACTIONS = ["dusuk", "orta", "yuksek"], ["kayit", "yonlendirme", "bilgi"] SCHEMA = {"talep_no": "sayi", "konu": TOPICS, "oncelik": PRIORITIES, "aksiyon": ACTIONS} PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"), ("faturami odedim ama sistemde borc gorunuyor", "fatura"), ("fatura ayrintisini nereden gorebilirim", "fatura"), ("evde su akmiyor bir ariza var galiba", "ariza"), ("baglanti bozuk oldu ve hic calismiyor", "ariza"), ("ariza kaydi actim ama ekip gelmedi", "ariza"), ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"), ("sayac degisimi icin randevu istiyorum", "sayac"), ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"), ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"), ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"), ("birim ucret abone tipine gore degisiyor mu", "tarife"), ("mahallede kesinti var ne zaman gelecek", "kesinti"), ("planli kesinti duyurusu nereden takip edilir", "kesinti"), ("aksamdan beri kesinti yasiyoruz", "kesinti")] TICKETS = [] for i in range(120): # 120 FICTIONAL support tickets r = rng(SEED + i) topic = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] TICKETS.append({"id": 4001 + i, "topic": topic, "text": [x for x in PATTERNS if x[1] == topic][int(r() * 3)][0]}) CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti"} TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} # misleading cues COUNTER = [0] def parse_record(text): """Key=value pair parser. Returns None for a response that cannot be parsed.""" if "=" not in text: return None record = {} for part in text.split(";"): if "=" not in part: return None key, _, value = part.partition("=") record[key.strip()] = value.strip() return record def validate(record): """Schema validation: missing key, extra key, invalid value.""" if record is None: return ["ayristirilamadi"] errors = [f"eksik:{k}" for k in SCHEMA if k not in record] errors += [f"fazla:{k}" for k in record if k not in SCHEMA] for key, rule in SCHEMA.items(): if key not in record: continue value = record[key] if rule == "sayi" and not value.isdigit(): errors.append(f"sayi_degil:{key}") elif rule != "sayi" and value not in rule: errors.append(f"gecersiz:{key}") return errors def draw(prompt, seed, salt): """One separately salted draw for each decision; decisions are not tied to each other.""" return rng((seed + salt) ^ trace_id(prompt))() def call(prompt, ticket, mode, seed=SEED): """mode: free | format | schema. In schema mode the generation cannot leave the schema.""" COUNTER[0] += 1 toks = tokenize(prompt) p = {k: 0.15 for k in TOPICS} for t in tokenize(ticket["text"]): # the topic is read only from the ticket text for table, w in ((CUES, 1.0), (TRAPS, 1.3)): if table.get(t) in p: p[table[t]] += w k = max(sorted(p), key=lambda a: p[a]) prio = PRIORITIES[pick(draw(prompt, seed, 11), [0.31, 0.44, 0.25])] act = ACTIONS[pick(draw(prompt, seed, 29), [0.46, 0.32, 0.22])] line = f"talep_no={ticket['id']} ; konu={k} ; oncelik={prio} ; aksiyon={act}" if mode == "schema": # schema imposition return line, k if mode == "free": return f"talebiniz {k} konusunda kaydedildi ve {act} adimi baslatildi", k tendency = 0.62 if "bicim" in toks else 0.0 # format description in the prompt if "ornek" in toks: tendency += 0.22 if draw(prompt, seed, 53) < tendency: return line, k broken = pick(draw(prompt, seed, 71), # response that does not follow the description [0.22, 0.20, 0.18, 0.16, 0.24]) if broken == 0: return f"talebiniz {k} konusunda kaydedildi ve {act} adimi baslatildi", k if broken == 1: return f"talep_no={ticket['id']} ; konu={k} ; aksiyon={act}", k if broken == 2: return f"talep_no={ticket['id']} ; konu={k} ; oncelik=acil ; aksiyon={act}", k if broken == 3: return f"talep_no=TL{ticket['id']} ; konu={k} ; oncelik={prio} ; aksiyon={act}", k return f"talep_no={ticket['id'] + 500} ; konu={k} ; oncelik={prio} ; aksiyon={act}", k FORMAT_DESCRIPTION = ("Yaniti su bicimde yaz: talep_no= ; konu= ; oncelik= ; aksiyon= . ") def build_prompt(prefix, x): return prefix + f"Talep {x['id']} : {x['text']}" print("schema keys:", " , ".join(SCHEMA)) for mode, prefix in (("free", "Su talebi yanitla . "), ("format", FORMAT_DESCRIPTION), ("schema", "Sema: talep_no ; konu ; oncelik ; aksiyon . ")): y, _ = call(build_prompt(prefix, TICKETS[0]), TICKETS[0], mode) errors = validate(parse_record(y)) print(f"mode {mode:<8} {y}") print(f" errors {errors if errors else 'none'}") print("\nfirst four validation errors in format description mode:") n = 0 for x in TICKETS: y, _ = call(build_prompt(FORMAT_DESCRIPTION, x), x, "format") errors = validate(parse_record(y)) if errors and n < 4: n += 1 print(f" {errors[0]:<22} {y[:52]}") ``` ``` schema keys: talep_no , konu , oncelik , aksiyon mode free talebiniz kesinti konusunda kaydedildi ve yonlendirme adimi baslatildi errors ['ayristirilamadi'] mode format talep_no=TL4001 ; konu=kesinti ; oncelik=orta ; aksiyon=kayit errors ['sayi_degil:talep_no'] mode schema talep_no=4001 ; konu=kesinti ; oncelik=orta ; aksiyon=bilgi errors none first four validation errors in format description mode: sayi_degil:talep_no talep_no=TL4001 ; konu=kesinti ; oncelik=orta ; aksi gecersiz:oncelik talep_no=4009 ; konu=sayac ; oncelik=acil ; aksiyon= ayristirilamadi talebiniz kesinti konusunda kaydedildi ve yonlendirm sayi_degil:talep_no talep_no=TL4032 ; konu=tarife ; oncelik=orta ; aksiy ``` The output that the three modes produced on the same ticket shows where the difference lies. The free text has the correct topic in it (`kesinti`, outage) but no record field; the parser finds nothing. In the format description mode the response is almost correct: all four fields are in place, but two letters have come before the ticket number and the field is no longer a number. The record system **rejects** this line, even though the content of the response is entirely correct. The four error examples also say where the breakage is: one falls because a field is not a number, one because it carries a value outside the closed set, one because it could not be parsed at all. All of them are format errors and none is a topic error. #### The Conformance and Cost Table of the Three Levels The same 120 tickets are run with four prompts. The first three ask **by writing into the prompt**, the last one constrains the generation. The second run is taken with seed 20260219 and the schema-conforming rate is printed separately for it. ```python EXAMPLE = "Ornek: talep_no=4000 ; konu=ariza ; oncelik=orta ; aksiyon=kayit . " LEVELS = [("free text", "free", "Su talebi yanitla . "), ("format description", "format", FORMAT_DESCRIPTION), ("format description + example", "format", FORMAT_DESCRIPTION + EXAMPLE), ("imposed schema", "schema", "Sema: talep_no ; konu ; oncelik ; aksiyon . ")] def read_topic(y, record): if record and "konu" in record: return record["konu"] return next((s for s in y.split() if s in TOPICS), None) def measure(mode, prefix, seed=SEED): COUNTER[0] = 0 parsed = valid = hits = recordable = n_in = n_out = 0 tally = {} for x in TICKETS: prompt = build_prompt(prefix, x) y, _ = call(prompt, x, mode, seed) record = parse_record(y) errors = validate(record) parsed += record is not None valid += not errors hit = read_topic(y, record) == x["topic"] hits += hit recordable += hit and not errors for e in errors: tally[e.split(":")[0]] = tally.get(e.split(":")[0], 0) + 1 n_in += len(tokenize(prompt)) n_out += len(tokenize(y)) return (parsed / 120, valid / 120, hits / 120, recordable / 120, n_in, n_out, tally) print("level parseable valid A valid B topic hit recordable input output") for name, mode, prefix in LEVELS: a, b = measure(mode, prefix), measure(mode, prefix, SEED + 1) print(f"{name:<28} {a[0]:>9.4f} {a[1]:>7.4f} {b[1]:>7.4f} {a[2]:>9.4f}" f" {a[3]:>10.4f} {a[4]:>5} {a[5]:>6}") for name, mode, prefix in LEVELS[1:3]: print(f"{name:<28} error distribution {dict(sorted(measure(mode, prefix)[6].items()))}") ``` ``` level parseable valid A valid B topic hit recordable input output free text 0.0000 0.0000 0.0000 0.8833 0.0000 2254 1614 format description 0.9333 0.8000 0.7167 0.8833 0.7000 5614 1789 format description + example 0.9667 0.8833 0.8667 0.8833 0.7750 7654 1789 imposed schema 1.0000 1.0000 1.0000 0.8833 0.8833 2974 1810 format description error distribution {'ayristirilamadi': 8, 'eksik': 1, 'gecersiz': 9, 'sayi_degil': 6} format description + example error distribution {'ayristirilamadi': 4, 'eksik': 5, 'gecersiz': 2, 'sayi_degil': 3} ``` The first row is a complete loss from the record system's point of view. The topic hit rate of the free text is **0.8833**, that is, about eight of every nine responses state the correct topic. The rate of recordable correct records, however, is **0.0000**. The correct information was produced and **ignored because it could not be read**. This is the clearest row showing why format checking is a criterion and not a decoration. The second row is the partial gain of writing into the prompt: parseable responses **0.9333**, but schema-conforming **0.8000**. The 0.1333 in between consists of responses that can be fully parsed but fail validation: in nine the priority is outside the closed set, in six the ticket number is not a number, in one a field is missing. Parseability and validity are not the same thing and cannot be measured with a single number. #### The Unmeasurable Gain and the Reverse Side of the Cost The third row is the one that fits intuition best and sits weakest in the table. Adding an example raises the conformance **from 0.8000 to 0.8833**; the gain is **0.0833**. The two-run columns of the format description row, which is the base of the comparison, need to be looked at: that row gives **0.7167** on the second seed, that is, its two-run difference is also **0.0833**. The gain is **not larger** than the difference. The rule of the course applies directly here: if the difference a prompt change adds does not exceed the two-run difference, it **counts as unmeasured**. Adding an example may have worked, but this table does not show it. The cost of the same change is certain, however. The input tokens go **from 5614 to 7654**: seventeen per ticket, **2040 tokens** in total. A measurable cost paid for an unmeasurable gain is the exact counterpart of the second claim of this course. The fourth row is the opposite of this. The imposed schema raises the conformance to **1.0000**; the gain over the base is **0.2000**, two and a half times the two-run difference. The cost column is even more surprising: the input tokens stay at **2974**. That is a little more than half of the format description and two fifths of the description with the example. This is because the schema is not **explained** to the prompt at length, it is declared briefly with the field names, and the real work is done in constraining the generation. The cheapest prompt gives the highest conformance. The output tokens, on the other hand, are close to each other at the four levels (between 1614 and 1810), because what changes is not the length of the output but its **format**. Finally, the topic hit rate column is fixed at **0.8833** on all four rows, and this is not a coincidence. The imposed schema guarantees the **format** of the response, not its **content**. Fourteen tickets are still assigned to the wrong topic, and this wrong topic now stands inside a perfectly formatted record, as a valid value. Validity is not correctness; a record that passes the schema can be wrong. Indeed, in the format description row, thirteen of the records that pass the schema carry a ticket number that is not on record instead of the number written into the prompt, and the validator sees none of them. #### Summary - The schema is four fields and its validator is written inside the lesson with the standard library; the error classes are counted separately. - In free text the topic hit rate is 0.8833 while the rate of recordable correct records is 0.0000: the correct information is ignored because it cannot be read. - The format description raises the conformance to 0.8000; parseability (0.9333) and conformance to the schema are not the same thing. - Adding an example raises the conformance to 0.8833, but the gain is 0.0833 and the two-run difference of the format description row is also 0.0833: the gain counts as unmeasured and 2040 tokens are paid for it. - The imposed schema raises the conformance to 1.0000 and does so with 2974 input tokens; the cheapest prompt gives the highest conformance. - The topic hit rate is fixed at 0.8833 at all four levels and thirteen records that pass the schema carry a ticket number that is not on record: the schema guarantees the format, not the content. #### Next Step In this lesson the validator was used only as a counter: it counted how many responses fell, and the fallen responses were thrown away. In a production pipeline throwing away is not an option; a fallen record is either recovered or processed by hand. Moreover, the problem seen in the last paragraph was never addressed: a record that passes the schema can violate a **business rule**, and a ticket number that is not on record is the clearest example. The next lesson counts the two checks separately and builds a retry loop on top: how many responses do not conform to the schema, how many get caught on a business rule, how many of these the retry recovers and how many extra calls this recovery costs. ### Output Validation Source: https://academia.sh/en/courses/prompt-engineering/output-validation Counting schema validation and business rule checking separately and building a retry loop on top: 24 of 120 fictional responses do not conform to the schema, 24 of the 96 records that pass the schema get caught on a business rule, and with no validation the faulty record rate is 0.4000. The two checks together bring the acceptance rate down to 0.6000, one retry recovers 26 records and raises it to 0.8167 at a cost of 48 extra calls. The second retry recovers eight more records but its gain of 0.0666 stays below the two-run difference of 0.0667. The retry that writes the error into the prompt recovers 23 records and adds 514 tokens. The previous lesson measured the format of the output on three levels and used the validator only as a counter: it counted how many responses fell and threw the fallen ones away. Two gaps were written into its closing. The first: in a production pipeline throwing away a fallen record is not an option. The second: a record that passes the schema can also be wrong. In the format description row of the previous lesson, thirteen of the records that passed the schema carried a ticket number that was not on record instead of the number written into the prompt, and the validator never saw it. This lesson separates the two checks and builds a retry loop on top. The procedure of input validation, schema and business rule was measured in the Web API Design course and in the Cybersecurity curriculum; it is not repeated here. The only thing added is **that the producer is uncertain**: a rejected record can be requested again, and the second request can give a different response. This is also what will be measured — what validation catches, what the retry recovers and how many calls it costs. - **OC25.** The code is a **simulator**; no real endpoint is called. The tickets, the fields and the rules are **fictional**, seed **20260218**. - **OC26.** The starting prompt is the **format description** prompt of the previous lesson; the imposed schema is deliberately not used, because this lesson measures what is done with an **erroneous output**. - **OC27. The two checks are separate.** Schema validation looks at the presence of the fields and their value sets. The **business rule check** is applied only to records that passed the schema and looks at the rules of the institution. - **OC28. Three business rules are fictional:** a high-priority record cannot be closed with an information action, an outage record cannot be closed with a record action alone, and the ticket number must be in the set of registered tickets. The schema **cannot see** any of the three. - **OC29. The retry repeats the same request with a new seed.** Because the model is uncertain, the second response can come out different; this is the only basis of the correction. - **OC30.** A variant **writes the error into the prompt**: the error name of the rejected response is added to the prompt. This raises the input tokens. - **OC31. A faulty record that enters the system** is counted separately: with no validation the acceptance rate looks like 1.0000, but a part of the records is broken. - **OC32.** The cost is written as the number of extra calls and total tokens; for the two-run difference the second run is taken with seed 20260219. #### The Error the Schema Cannot See A schema defines the **format** of a record: which fields exist and which set the values come from. The rules of an institution, on the other hand, look at the **meaning** of the record and cannot be written in the language of a schema. This lesson defines three fictional rules and shows that all three are triggered on perfectly formatted records. ```python # SIMULATOR -- FICTIONAL. No real model endpoint is called. The schema validation and the # business rule check are written inside the lesson; no library is used from outside. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): # FICTIONAL tokenizer, a piece has at most five characters out = [] for piece in text.lower().replace(",", " , ").split(): while len(piece) > 5: out.append(piece[:5]) piece = piece[5:] out.append(piece) return out def trace_id(text): h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] PRIORITIES, ACTIONS = ["dusuk", "orta", "yuksek"], ["kayit", "yonlendirme", "bilgi"] SCHEMA = {"talep_no": "sayi", "konu": TOPICS, "oncelik": PRIORITIES, "aksiyon": ACTIONS} PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"), ("faturami odedim ama sistemde borc gorunuyor", "fatura"), ("fatura ayrintisini nereden gorebilirim", "fatura"), ("evde su akmiyor bir ariza var galiba", "ariza"), ("baglanti bozuk oldu ve hic calismiyor", "ariza"), ("ariza kaydi actim ama ekip gelmedi", "ariza"), ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"), ("sayac degisimi icin randevu istiyorum", "sayac"), ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"), ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"), ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"), ("birim ucret abone tipine gore degisiyor mu", "tarife"), ("mahallede kesinti var ne zaman gelecek", "kesinti"), ("planli kesinti duyurusu nereden takip edilir", "kesinti"), ("aksamdan beri kesinti yasiyoruz", "kesinti")] TICKETS = [] for i in range(120): # 120 FICTIONAL support tickets r = rng(SEED + i) topic = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] TICKETS.append({"id": 4001 + i, "topic": topic, "text": [x for x in PATTERNS if x[1] == topic][int(r() * 3)][0]}) CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti"} TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} # misleading cues COUNTER = [0] def parse_record(text): """Key=value pair parser. Returns None for a response that cannot be parsed.""" if "=" not in text: return None record = {} for part in text.split(";"): if "=" not in part: return None key, _, value = part.partition("=") record[key.strip()] = value.strip() return record def validate(record): """Schema validation: missing key, extra key, invalid value.""" if record is None: return ["ayristirilamadi"] errors = [f"eksik:{k}" for k in SCHEMA if k not in record] errors += [f"fazla:{k}" for k in record if k not in SCHEMA] for key, rule in SCHEMA.items(): if key not in record: continue value = record[key] if rule == "sayi" and not value.isdigit(): errors.append(f"sayi_degil:{key}") elif rule != "sayi" and value not in rule: errors.append(f"gecersiz:{key}") return errors def draw(prompt, seed, salt): """One separately salted draw for each decision; decisions are not tied to each other.""" return rng((seed + salt) ^ trace_id(prompt))() def call(prompt, ticket, seed=SEED): """The format description prompt of the previous lesson; a share of the responses do not follow it.""" COUNTER[0] += 1 toks = tokenize(prompt) p = {k: 0.15 for k in TOPICS} for t in tokenize(ticket["text"]): # the topic is read only from the ticket text for table, w in ((CUES, 1.0), (TRAPS, 1.3)): if table.get(t) in p: p[table[t]] += w k = max(sorted(p), key=lambda a: p[a]) prio = PRIORITIES[pick(draw(prompt, seed, 11), [0.31, 0.44, 0.25])] act = ACTIONS[pick(draw(prompt, seed, 29), [0.46, 0.32, 0.22])] line = f"talep_no={ticket['id']} ; konu={k} ; oncelik={prio} ; aksiyon={act}" tendency = 0.62 if "bicim" in toks else 0.0 # format description in the prompt if "hatal" in toks: # retry that writes the error into the prompt tendency += 0.14 if draw(prompt, seed, 53) < tendency: return line broken = pick(draw(prompt, seed, 71), # response that does not follow the description [0.22, 0.20, 0.18, 0.16, 0.24]) if broken == 0: return f"talebiniz {k} konusunda kaydedildi ve {act} adimi baslatildi" if broken == 1: return f"talep_no={ticket['id']} ; konu={k} ; aksiyon={act}" if broken == 2: return f"talep_no={ticket['id']} ; konu={k} ; oncelik=acil ; aksiyon={act}" if broken == 3: return f"talep_no=TL{ticket['id']} ; konu={k} ; oncelik={prio} ; aksiyon={act}" return f"talep_no={ticket['id'] + 500} ; konu={k} ; oncelik={prio} ; aksiyon={act}" FORMAT_DESCRIPTION = ("Yaniti su bicimde yaz: talep_no= ; konu= ; oncelik= ; aksiyon= . ") def build_prompt(prefix, x): return prefix + f"Talep {x['id']} : {x['text']}" REGISTERED = {t["id"] for t in TICKETS} def business_rules(record): """Institutional rules of a record that passed the schema. The schema sees none of them.""" errors = [] if record.get("oncelik") == "yuksek" and record.get("aksiyon") == "bilgi": errors.append("yuksek_oncelik_bilgi_ile_kapanamaz") if record.get("konu") == "kesinti" and record.get("aksiyon") == "kayit": errors.append("kesinti_kayitla_kapanamaz") if record.get("talep_no", "").isdigit() and int(record["talep_no"]) not in REGISTERED: errors.append("kayitli_olmayan_talep_no") return errors schema_fail, rule_fail, seen = 0, 0, {} for x in TICKETS: y = call(build_prompt(FORMAT_DESCRIPTION, x), x) record = parse_record(y) errors = validate(record) if errors: schema_fail += 1 continue rules = business_rules(record) rule_fail += bool(rules) if rules: seen.setdefault(rules[0], y) print(f"FICTIONAL tickets {len(TICKETS)}: caught by schema {schema_fail}, " f"caught by business rule {rule_fail}") for e, y in sorted(seen.items()): print(f" {e}\n {y}") ``` ``` FICTIONAL tickets 120: caught by schema 24, caught by business rule 24 kayitli_olmayan_talep_no talep_no=4505 ; konu=sayac ; oncelik=orta ; aksiyon=kayit kesinti_kayitla_kapanamaz talep_no=4015 ; konu=kesinti ; oncelik=dusuk ; aksiyon=kayit yuksek_oncelik_bilgi_ile_kapanamaz talep_no=4022 ; konu=tarife ; oncelik=yuksek ; aksiyon=bilgi ``` The two numbers are equal to each other, and this equality is the starting point of the lesson: **24 responses do not conform to the schema, and 24 of the 96 responses that pass the schema are caught on a business rule.** That is, the format check sees only half of the errors. All three examples are perfectly formatted; the parser and the validator find nothing in any of them. In the first the ticket number is a number not written in the prompt, in the second an outage record has been closed with the wrong action, in the third a high-priority record has been tied to an information action. These three rules cannot be translated into the language of the schema, because none of them concerns the value of a single field. The first two are a consistency **between fields**, the third is the consistency of the record **with the outside world**. Without counting these two classes separately in the output of an uncertain producer, validation does not count as measured. #### What the Retry Recovers Six pipelines are run on the same 120 tickets. The first three have no retry and only measure what the check eliminates; the last three request the rejected record again. ```python PIPELINES = [("no validation", False, False, 0, False), ("schema only", True, False, 0, False), ("schema + business rule", True, True, 0, False), ("+ 1 retry", True, True, 1, False), ("+ 2 retries", True, True, 2, False), ("+ 1 retry, error in prompt", True, True, 1, True)] def check(record, with_schema, with_rules): errors = validate(record) if with_schema else [] if not errors and with_rules and record is not None: errors = business_rules(record) return errors def run_pipeline(with_schema, with_rules, retries, write_error, seed=SEED): COUNTER[0] = 0 first_schema = first_rule = recovered = accepted = faulty = extra = tok = 0 for x in TICKETS: prompt = build_prompt(FORMAT_DESCRIPTION, x) y = call(prompt, x, seed) tok += len(tokenize(prompt)) + len(tokenize(y)) record = parse_record(y) first_schema += bool(validate(record)) first_rule += not validate(record) and bool(business_rules(record)) errors = check(record, with_schema, with_rules) for attempt in range(retries): if not errors: break extra += 1 prompt2 = prompt + f" Onceki yanit hatali: {errors[0]} ." if write_error else prompt y = call(prompt2, x, seed + 1 + attempt) tok += len(tokenize(prompt2)) + len(tokenize(y)) record = parse_record(y) errors = check(record, with_schema, with_rules) recovered += not errors accepted += not errors faulty += (not errors) and bool(validate(record) or business_rules(record)) return (first_schema, first_rule, recovered, accepted / 120, faulty / 120, extra, tok) print("pipeline schema rule recovered accepted faulty extra calls tokens") for name, s, r, n, w in PIPELINES: a = run_pipeline(s, r, n, w) print(f"{name:<26} {a[0]:>6} {a[1]:>4} {a[2]:>9} {a[3]:>8.4f} {a[4]:>6.4f}" f" {a[5]:>11} {a[6]:>6}") b = run_pipeline(True, True, 1, False, SEED + 1) print(f"second run (+ 1 retry): accepted {b[3]:.4f}, recovered {b[2]}, " f"extra calls {b[5]}") ``` ``` pipeline schema rule recovered accepted faulty extra calls tokens no validation 24 24 0 1.0000 0.4000 0 7403 schema only 24 24 0 0.8000 0.2000 0 7403 schema + business rule 24 24 0 0.6000 0.0000 0 7403 + 1 retry 24 24 26 0.8167 0.0000 48 10350 + 2 retries 24 24 34 0.8833 0.0000 70 11716 + 1 retry, error in prompt 24 24 23 0.7917 0.0000 48 10864 second run (+ 1 retry): accepted 0.7500, recovered 26, extra calls 56 ``` The first three rows were produced with a single call each, and the token column is the same in all three: **7403**. The only thing that changes is the check made before acceptance. With no validation the acceptance rate looks like **1.0000** and the rate of wrong records entering the record system is **0.4000**. That is, when nothing is measured the system looks flawless, and two of every five records are broken. When the schema is added the faulty records fall to **0.2000**, and when the business rule is added as well, to **0.0000**. The price of this is the acceptance rate: **from 1.0000 to 0.6000**. Validation corrects no record; it only makes them visible. The system in the third row is better than the first but does less work; 48 tickets are left on hand. The measure of validation is not the acceptance rate but **how much of what is accepted is correct**. #### The Cost of Recovery The fourth row adds the retry. **26 records are recovered** and the acceptance rate rises from 0.6000 to **0.8167**; the faulty record rate stays at **0.0000**, because the recovery has been placed behind the check, not in front of it. The cost is **48 extra calls** and **2947 tokens**: 1.85 calls and 113.3 tokens per recovered record. Since the gain is 0.2167 and the two-run difference is 0.0667, this row is comfortably measurable. The fifth row is where the rule of the course is applied most sharply. The second retry recovers eight more records and raises the acceptance rate to **0.8833**. The gain is **0.0666**. The two-run difference of the previous pipeline is **0.0667**: 0.8167 in the first run, 0.7500 in the second. The gain is **not larger** than the difference, so it **counts as unmeasured**. The cost paid for this row, on the other hand, is certain: 22 extra calls and 1366 tokens, 2.75 calls per recovered record. An unmeasurable gain has been bought with a measurable cost. The sixth row tests the correction that best fits intuition: writing the error of the rejected response into the prompt. The result is **23 recoveries** instead of 26 and **0.7917** in the acceptance rate. Since the 0.0250 in between stays far below the two-run difference of 0.0667, there is **no measurable difference** between this row and the fourth. The cost is measurable, however: with the same 48 calls the tokens rise from 10350 to **10864**, that is, **514 tokens** have been added. Writing the errors into the prompt is here not an improvement but an unmeasured guess. It is wrapped up with three numbers. The previous prompt is the format description prompt: acceptance 0.6000, faulty record 0.0000, 7403 tokens, 120 calls. The changed pipeline adds one retry: acceptance 0.8167, the same faulty record rate, 10350 tokens, 168 calls. The cost is **48 calls and 2947 tokens**; for the first time in this course a scaffolding moves the **number of calls**, and the larger part of the cost sits there. #### Summary - Schema validation eliminates 24 responses, the business rule check eliminates 24 more of the 96 responses that pass the schema: the format check sees half of the errors. - All three business rules are triggered on perfectly formatted records; two are consistency between fields, one is the consistency of the record with the outside world. - With no validation the acceptance rate is 1.0000 and the faulty record entering the system is 0.4000; the two checks bring the faulty record down to 0.0000 while lowering the acceptance to 0.6000. Validation does not correct, it makes visible. - One retry recovers 26 records and raises the acceptance to 0.8167; its cost is 48 extra calls and 2947 tokens, 1.85 calls per recovered record. - The gain of the second retry is 0.0666 and the two-run difference of the pipeline is 0.0667: the gain counts as unmeasured and 22 calls and 1366 tokens are paid for it. - Writing the error into the prompt recovers 23 records instead of 26 and adds 514 tokens; there is no measurable difference between the two pipelines. #### Next Step In this lesson the recovery was done with a single lever: asking the same request again. The 14 records that could not be recovered were never examined. Some of them will not be corrected however much is tried, because the problem is not in the draw of the response but in **something the model does not know**. If the tariff tier a ticket falls into is not written into the prompt, a better instruction cannot produce it. The next lesson counts this distinction: the same wrong response is sometimes corrected by a better instruction and sometimes only by supplying the missing information. Which error is corrected by which is counted separately and the cost of the two is compared. ### Prompt and Context Engineering Difference Source: https://academia.sh/en/courses/prompt-engineering/prompt-and-context-engineering-difference Comparing two separate fixes on the same set of errors: on 120 fictional tickets the baseline topic hit rate is 0.8833, the fault code hit rate is 0.4083, and both correct is 0.4083. Writing a distinguishing rule into the prompt raises the topic hit rate to 0.9667 and does not touch the code hit rate at all; putting the fictional code table into the prompt raises the code hit rate to 0.9167 and does not touch the topic hit rate at all. Of the fourteen topic errors ten are fixed by the instruction alone, of the seventy-one code errors sixty-six are fixed by information alone, and one error in each class is fixed by neither. The costs diverge: the instruction adds 1680 tokens, the information adds 14520. The previous lesson recovered the rejected records by asking again, and fourteen records were not fixed by any retry. This lesson asks the cause of that residue. A wrong response can be wrong for two separate reasons, and the two do not resemble each other. The model may have misunderstood **what to do**: the instruction is ambiguous, a misleading word has pulled the decision. Or the model may have been forced to say **something it does not know**: the requested information is not in the prompt at all and no instruction can produce it. The first class is the subject of prompt design. The second is the subject of **context engineering**: not what the model is told, but what is placed next to it. This lesson only establishes the distinction and measures it. Which document is found where, how it is retrieved and how many pieces are selected is a retrieval pipeline question and is left to the Embeddings and Retrieval-Augmented Generation course; here the fictional table is put into the prompt **as it is**. - **OC33.** The code is a **simulator**; no real endpoint is called. The tickets, the fault codes and the code table are **fictional**, seed **20260218**. - **OC34. The task has two fields:** assigning a topic and a **fault code** to the ticket. The two fields carry two separate sources of error. - **OC35. The source of the topic error is misleading cues.** A word in a side clause of the ticket pulls the decision to another topic. This is an instruction problem. - **OC36. The source of the code error is missing information.** When the code table is not in the prompt, the simulator writes the **most common code** of the topic; only five of the fifteen codes can be found this way. - **OC37. The instruction fix** writes a distinguishing rule into the prompt. The rule is not always followed; the simulator applies it in a share of the responses. - **OC38. The information fix** puts the whole fictional code table into the prompt. **No retrieval is done**: which row is needed is not looked up, the table is given in full. The model matches the table against its own definition texts and reads the wrong row in a share of the responses. - **OC39.** An error is a wrong value in a field; errors are counted **per field**. The same record can carry both errors at once. - **OC40.** The cost is written as input tokens and also given as **tokens per fixed error**; the second seed for the two runs is 20260219. #### Two Errors, Two Separate Sources Two example tickets are put side by side in four scaffoldings (`SETUPS` in the code). The first carries both errors at once, the second only the code error. ```python # SIMULATOR -- FICTIONAL. No real model endpoint is called. The code table is also # FICTIONAL and is put into the prompt as it is, without any retrieval. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): # FICTIONAL tokenizer, a piece has at most five characters out = [] for piece in text.lower().replace(",", " , ").split(): while len(piece) > 5: out.append(piece[:5]) piece = piece[5:] out.append(piece) return out def trace_id(text): h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"), ("faturami odedim ama sistemde borc gorunuyor", "fatura"), ("fatura ayrintisini nereden gorebilirim", "fatura"), ("evde su akmiyor bir ariza var galiba", "ariza"), ("baglanti bozuk oldu ve hic calismiyor", "ariza"), ("ariza kaydi actim ama ekip gelmedi", "ariza"), ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"), ("sayac degisimi icin randevu istiyorum", "sayac"), ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"), ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"), ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"), ("birim ucret abone tipine gore degisiyor mu", "tarife"), ("mahallede kesinti var ne zaman gelecek", "kesinti"), ("planli kesinti duyurusu nereden takip edilir", "kesinti"), ("aksamdan beri kesinti yasiyoruz", "kesinti")] TICKETS = [] for i in range(120): # 120 FICTIONAL support tickets r = rng(SEED + i) topic = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] TICKETS.append({"id": 4001 + i, "topic": topic, "text": [x for x in PATTERNS if x[1] == topic][int(r() * 3)][0]}) CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti"} TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} # misleading cues CODES = [("FT-21", "fatura tutari yuksek geldi"), ("FT-34", "odenen fatura borc gorunuyor"), ("FT-47", "fatura ayrinti talebi"), ("AR-12", "evde su akmiyor sikayeti"), ("AR-25", "baglanti bozuk calismiyor"), ("AR-38", "acik kayit ekip gelmedi"), ("SY-16", "endeks yanlis girilmis"), ("SY-29", "sayac degisim randevusu"), ("SY-42", "endeks ile fatura uyusmuyor"), ("TR-13", "tarife basamagi sorusu"), ("TR-26", "tarife degisiklik talebi"), ("TR-39", "birim ucret abone tipi"), ("KS-15", "mahalle kesintisi suresi"), ("KS-28", "planli duyuru takibi"), ("KS-41", "aksamdan beri kesinti")] TRUE_CODE = {t: CODES[i][0] for i, (t, _) in enumerate(PATTERNS)} TOPIC_CODE = {} for i, (t, k) in enumerate(PATTERNS): TOPIC_CODE.setdefault(k, CODES[i][0]) # first code of the topic when there is no document DOCUMENT = " ; ".join(f"{k} {a}" for k, a in CODES) COUNTER = [0] def draw(prompt, seed, salt): """One separately salted draw for each decision; decisions are not tied to each other.""" return rng((seed + salt) ^ trace_id(prompt))() def call(prompt, ticket, seed=SEED): """Topic and fault code are requested together. When the code table is not in the prompt the model writes the most common code of the topic; when it is, it chooses by overlap with the definitions.""" COUNTER[0] += 1 toks = tokenize(prompt) follows = "ayirt" in toks and draw(prompt, seed, 91) < 0.78 trap_weight = 0.9 if follows else 1.3 # distinguishing rule in the prompt p = {k: 0.15 for k in TOPICS} for t in tokenize(ticket["text"]): for table, w in ((CUES, 1.0), (TRAPS, trap_weight)): if table.get(t) in p: p[table[t]] += w topic = max(sorted(p), key=lambda a: p[a]) if "tablo" not in toks: # no code table in the prompt return topic, TOPIC_CODE[topic] tb = set(tokenize(ticket["text"])) scores = [(len(tb & set(tokenize(a))), -i) for i, (_, a) in enumerate(CODES)] i = -max(scores)[1] if draw(prompt, seed, 17) < 0.08: # chance of misreading the document i = (i + 1) % len(CODES) return topic, CODES[i][0] RULE_PROMPT = "Ayirt edici kural: yan cumledeki sozcukler etiketi belirlemez . " TABLE_PROMPT = "Ariza kodu tablosu: " + DOCUMENT + " . " print(f"FICTIONAL code table {len(CODES)} rows, {len(tokenize(DOCUMENT))} tokens") print("sample rows:", " | ".join(f"{k} {a}" for k, a in CODES[3:6])) for j in (3, 0): x = TICKETS[j] print(f"\nticket {x['id']} : {x['text']}") print(f" true topic {x['topic']:<8} code {TRUE_CODE[x['text']]}") for name, prefix in (("baseline", ""), ("instruction", RULE_PROMPT), ("information", TABLE_PROMPT), ("both", RULE_PROMPT + TABLE_PROMPT)): topic, code = call(prefix + f"Talep {x['id']} : {x['text']}", x) mark = "correct" if (topic == x["topic"] and code == TRUE_CODE[x["text"]]) else "" print(f" {name:<13} topic {topic:<8} code {code} {mark}") ``` ``` FICTIONAL code table 15 rows, 116 tokens sample rows: AR-12 evde su akmiyor sikayeti | AR-25 baglanti bozuk calismiyor | AR-38 acik kayit ekip gelmedi ticket 4004 : ariza kaydi actim ama ekip gelmedi true topic ariza code AR-38 baseline topic kesinti code KS-15 instruction topic ariza code AR-12 information topic kesinti code AR-38 both topic ariza code AR-38 correct ticket 4001 : planli kesinti duyurusu nereden takip edilir true topic kesinti code KS-28 baseline topic kesinti code KS-15 instruction topic kesinti code KS-15 information topic kesinti code KS-28 correct both topic kesinti code KS-28 correct ``` The upper ticket carries both errors at once. The topic comes out as `kesinti` (outage), because the last two words of the ticket trigger a misleading cue; when the distinguishing rule is written, the topic returns to `ariza` (fault). But on the same row the code is still wrong: `AR-12`, where the correct one is `AR-38`. The instruction fixed the topic and could not fix the code, because the correct code was written nowhere in the prompt. When the table is put in the code becomes `AR-38`, and this time the topic stays wrong. The record becomes correct only **with both together**. The topic of the lower ticket is correct from the start and only its code is wrong. The instruction row is identical to the baseline: the topic stays `kesinti`, the code stays `KS-15`. A rule of fourteen tokens was added to the prompt and no field of the output changed. The table, on the other hand, brings the code to `KS-28` in a single step. The same error form, two separate fixes — and the cost of choosing the wrong one is only tokens, with no gain at all. What the two examples say together is this: looking at a wrong response does not reveal which scaffolding is needed. The wrongness itself looks the same in both cases. The distinction appears only when **each of the two scaffoldings is tried and counted separately**. #### Which Error Is Fixed by Which The same 120 tickets are run in four scaffoldings and errors are counted **per field**. ```python SETUPS = [("baseline", ""), ("+ instruction", RULE_PROMPT), ("+ information", TABLE_PROMPT), ("+ both", RULE_PROMPT + TABLE_PROMPT)] def measure(prefix, seed=SEED): COUNTER[0] = 0 d, g = [], 0 for x in TICKETS: prompt = prefix + f"Talep {x['id']} : {x['text']}" topic, code = call(prompt, x, seed) d.append((topic == x["topic"], code == TRUE_CODE[x["text"]])) g += len(tokenize(prompt)) return d, g RESULTS = {name: measure(prefix) for name, prefix in SETUPS} print("setup topic hit rate code hit rate both correct input tokens") for name, _ in SETUPS: d, g = RESULTS[name] print(f"{name:<14} {sum(a for a, b in d) / 120:>14.4f} {sum(b for a, b in d) / 120:>13.4f}" f" {sum(a and b for a, b in d) / 120:>12.4f} {g:>12}") print("\nerror class baseline errors by instruction by information by both by neither") for name, j in (("topic error", 0), ("code error", 1)): n = [i for i in range(120) if not RESULTS["baseline"][0][i][j]] y = sum(RESULTS["+ instruction"][0][i][j] for i in n) b = sum(RESULTS["+ information"][0][i][j] for i in n) ik = sum(RESULTS["+ both"][0][i][j] for i in n) h = sum(not (RESULTS["+ instruction"][0][i][j] or RESULTS["+ information"][0][i][j] or RESULTS["+ both"][0][i][j]) for i in n) print(f"{name:<13} {len(n):>15} {y:>14} {b:>14} {ik:>7} {h:>10}") b1 = measure(RULE_PROMPT + TABLE_PROMPT, SEED + 1)[0] print(f"\nsecond run (+ both): topic {sum(a for a, _ in b1) / 120:.4f}, " f"code {sum(b for _, b in b1) / 120:.4f}, both correct " f"{sum(a and b for a, b in b1) / 120:.4f}") ``` ``` setup topic hit rate code hit rate both correct input tokens baseline 0.8833 0.4083 0.4083 1534 + instruction 0.9667 0.4083 0.4083 3214 + information 0.8833 0.9167 0.8000 16054 + both 0.9583 0.9000 0.8667 17734 error class baseline errors by instruction by information by both by neither topic error 14 10 0 9 1 code error 71 0 66 63 1 second run (+ both): topic 0.9667, code 0.8917, both correct 0.8583 ``` The second and third rows of the first table show the distinction in the plainest way. The instruction raises the topic hit rate **from 0.8833 to 0.9667** and leaves the code hit rate **at 0.4083** as it was. The information raises the code hit rate **from 0.4083 to 0.9167** and leaves the topic hit rate **at 0.8833** as it was. The two scaffoldings work on the same task but **never touch the same error**. The second table counts this per error. Of the fourteen topic errors **ten** are fixed by the instruction and **zero** by the information. Of the seventy-one code errors **sixty-six** are fixed by the information and **zero** by the instruction. One of the two columns is always the zero of the other; this is the numerical counterpart of the fact that the two fixes cannot substitute for each other. The first step in improving a system is counting which error is in which class — because every token spent on the wrong class is wasted. **One error** in each class is fixed by no scaffolding at all. On the topic side this is a call in which the distinguishing rule is not followed; on the code side it is a call in which the wrong row of the table is read. Both are the residue not of prompt design but of the uncertainty of the producer, and they do not close by writing more text. #### The Cost of the Two Fixes The cost column separates the two fixes sharply. The instruction raises the input tokens **from 1534 to 3214**: fourteen per ticket, **1680 tokens** in total. The information raises them to **16054**: one hundred twenty-one per ticket, **14520 tokens** in total and **10.5 times** the baseline. This is the cost of putting a whole code table into every request. Read per fixed error the picture softens but does not reverse: the instruction fixes ten errors with 1680 tokens, **168 tokens per error**; the information fixes sixty-six errors with 14520 tokens, **220 tokens per error**. The information is more expensive but **in return fixes more errors than can be measured against it**: the baseline carried 71 code errors and none of them would have been fixed however much text was added to the instruction. The last row also shows the cost of the two scaffoldings working together. The rate of records correct in both fields rises **from 0.4083 to 0.8667**; in the second run the same number is **0.8583**, so the two-run difference is **0.0084**. Since the gain is more than fifty times this difference, it is comfortably measurable. On the other hand the combined scaffolding is **a little worse in every field** than the single scaffoldings: topic 0.9583 instead of 0.9667, code 0.9000 instead of 0.9167. As the prompt grows, the share of calls in which the rule is followed and the table is read correctly falls. **A prompt changed without measurement is not an improvement but a guess**, and the sum of two improvements is not the sum of the two. #### Summary - A wrong response comes either from the model misunderstanding **what to do** or from its being forced to say **something it does not know**; the two are fixed by separate scaffoldings. - The instruction raises the topic hit rate from 0.8833 to 0.9667 and does not touch the code hit rate at all; the information raises the code hit rate from 0.4083 to 0.9167 and does not touch the topic hit rate at all. - Of the fourteen topic errors ten are fixed by the instruction alone, of the seventy-one code errors sixty-six are fixed by information alone; the reciprocal value of both columns is zero. - One error in each class is fixed by no scaffolding: these are the residue of the producer's uncertainty. - The instruction adds 1680 tokens and the information adds 14520; 168 and 220 tokens per fixed error. The information is expensive but fixes the errors the instruction never reaches. - The rate of records correct in both fields rises from 0.4083 to 0.8667, but the combined scaffolding is a little worse in every field than the single scaffoldings. #### Next Step In this lesson the errors were split into two sources and both were measured on single records. What was not asked is **who the errors are concentrated on**. A system can be 0.8667 correct and all of the remaining wrong answers can be concentrated in a particular set of subscribers; in that case the average is good and for that set the system is not working. The next lesson measures this: how far the subgroup gap can be narrowed with a prompt, what part of it remains, and why a prompt does not solve **data bias** is shown with numbers. ### Bias-Reducing Prompts Source: https://academia.sh/en/courses/prompt-engineering/bias-reducing-prompts Measuring how much of a subgroup gap prompt balancing closes and what it leaves: on 200 fictional tickets the central hit rate is 0.9016, the rural hit rate 0.5513 and the gap 0.3504. A bias-reducing notice in the prompt moves the gap to 0.3586 and adds 3800 tokens; a balanced example brings it down to 0.2888 and adds 14400 tokens. The hit rate of the sampled local spelling rises from 0.4848 to 0.8788, while the unsampled one stays fixed at 0.3023. The prompt closes only a 0.1758 share of the gap. The course wrap-up gathers the previous prompt, changed prompt and cost rows of twenty-two lessons. The previous lesson split the errors into two sources. What went unasked was **who the errors were concentrated on**: the overall hit rate can be acceptable while all the remaining wrong answers accumulate in one set of subscribers; for that set the system does not work, and the average hides it. Subgroup, gap and fairness measures were established in the Model Evaluation, Interpretability and Ethics course and are not repeated. This lesson's question is narrow: **how much of a subgroup gap can a prompt close, and what part remains.** - **OC41.** The code is a **simulator**; no real endpoint is called. Subscriber groups, ticket spellings and the cue table are **fictional**, seed **20260218**. - **OC42.** There are two fictional subscriber groups, **central** and **rural**; the group name never enters the decision and serves only measurement. Each topic has one **standard** and two **local** spellings; central subscribers mostly write the standard one, rural subscribers mostly the local ones. - **OC43. The source of the bias is in the data:** the cue table was built only from the **standard spelling**, and when no cue is found the decision **falls to the most frequent class**. - **OC44. The first balancer** is a notice written into the prompt (the spelling does not affect the decision), which in the simulator turns the fall into an equal draw. **The second balancer** is the balanced example: each topic's **local A** spelling goes into the prompt with its label, and the simulator extracts a temporary cue table from the examples. **Local B is not sampled.** - **OC45.** The measures are the topic hit rate per group and the **gap** (central minus rural, small is good). Cost is written as input tokens; the second run has seed 20260219. #### The Source of the Gap The gap does not come from the model knowing something about the groups; the group name is never in the prompt. Its source is **which texts built the cue table**. ```python # SIMULATOR -- FICTIONAL. No real model endpoint is called. Subscriber groups, # ticket spellings and the cue table are FICTIONAL. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): # FICTIONAL tokenizer, a piece has at most five characters out = [] for piece in text.lower().replace(",", " , ").split(): while len(piece) > 5: out.append(piece[:5]) piece = piece[5:] out.append(piece) return out def trace_id(text): h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h # each topic: one standard and two local spellings; the local ones are FICTIONAL SPELLINGS = { "fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "bu ayki hesap pusulasi baya kabarik cikmis", "gecen ayki odemeye gore rakam iki misli olmus"], "ariza": ["evde su akmiyor bir ariza var galiba", "muslugu actim damla bile gelmiyor", "avludaki boru patlamis her yer golet"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "saatin rakamlari bizimkiyle uyusmuyor", "kutunun icindeki numara yerinde saymis"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "bizden hangi ucretlendirmeyle para aliniyor", "kademeli hesaplama nasil yapiliyor anlamadim"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "iki gundur su yok kimse haber vermedi", "koyun ustunde bir gundur akmiyor diyorlar"], } TOPICS = list(SPELLINGS) # central mostly write the standard spelling, rural mostly the local ones SPELLING_SHARE = {"merkez": [0.86, 0.08, 0.06], "kirsal": [0.24, 0.40, 0.36]} TICKETS = [] for i in range(200): r = rng(SEED + i) group = "merkez" if r() < 0.62 else "kirsal" topic = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] j = pick(r(), SPELLING_SHARE[group]) TICKETS.append({"id": 4001 + i, "topic": topic, "group": group, "spelling": j, "text": SPELLINGS[topic][j]}) # the cue table was built ONLY from the standard spelling: this is the data bias CUES = {} for k in TOPICS: for t in tokenize(SPELLINGS[k][0]): CUES.setdefault(t, k) MOST_COMMON = "fatura" # with no evidence it falls to the most frequent class COUNTER = [0] def example_table(toks): """Temporary cue table from the 'example: -> ' pieces.""" table, buffer = {}, [] for i, t in enumerate(toks): if t == "->" and i + 1 < len(toks) and toks[i + 1] in TOPICS: for s in buffer: table.setdefault(s, toks[i + 1]) buffer = [] elif t == "ornek": buffer = [] else: buffer.append(t) return table def call(prompt, ticket, seed=SEED): """Topic assignment. Without evidence it falls to the most frequent class; the bias-reducing notice turns this fall into an equal draw.""" COUNTER[0] += 1 toks = tokenize(prompt) table = dict(CUES) table.update(example_table(toks)) # examples in the prompt extend the table p = {k: 0.0 for k in TOPICS} for t in tokenize(ticket["text"]): if table.get(t) in p: p[table[t]] += 1.0 if max(p.values()) == 0.0: # no cue at all if "esit" in toks: # bias-reducing notice return TOPICS[pick(rng((seed + 5) ^ trace_id(prompt))(), [0.2] * 5)] return MOST_COMMON return max(sorted(p), key=lambda a: p[a]) NOTICE = ("Talebin yazim bicimi karari etkilemez ; her yazim esit olcutle " "degerlendirilir . ") EXAMPLES = " ".join(f"ornek: {SPELLINGS[k][1]} -> {k} ." for k in TOPICS) + " " print(f"FICTIONAL tickets {len(TICKETS)}: central {sum(x['group'] == 'merkez' for x in TICKETS)}, " f"rural {sum(x['group'] == 'kirsal' for x in TICKETS)}") local_count = {g: sum(x["group"] == g and x["spelling"] > 0 for x in TICKETS) for g in ("merkez", "kirsal")} print(f"tickets with local spelling: central {local_count['merkez']}, " f"rural {local_count['kirsal']}") print(f"cue table {len(CUES)} entries, all from the standard spelling") for j, name in enumerate(("standard", "local A", "local B")): x = next(t for t in TICKETS if t["topic"] == "sayac" and t["spelling"] == j) print(f" {name:<9} {x['text'][:42]:<43} -> {call('Talep: ' + x['text'], x)}") ``` ``` FICTIONAL tickets 200: central 122, rural 78 tickets with local spelling: central 18, rural 58 cue table 49 entries, all from the standard spelling standard sayac okumasi yanlis girilmis endeks tutmu -> sayac local A saatin rakamlari bizimkiyle uyusmuyor -> fatura local B kutunun icindeki numara yerinde saymis -> fatura ``` Three spellings of one topic give three results. The standard spelling is read correctly; both local spellings come out as `fatura` (invoice), because no cue is found in the table and the decision falls to the most frequent class. 58 of the rural tickets are local, only 18 of the central ones; the error accumulates on rural subscribers. The model does **not distinguish** them — it only fails to recognize a text with no counterpart in its own table, and who wrote that text is a fact from the data. #### What the Prompt Closes and What It Does Not ```python SETUPS = [("baseline", ""), ("+ notice", NOTICE), ("+ balanced example", EXAMPLES), ("+ both", NOTICE + EXAMPLES)] def measure(prefix, seed=SEED): COUNTER[0] = 0 by_group = {"merkez": [0, 0], "kirsal": [0, 0]} by_spelling = [[0, 0], [0, 0], [0, 0]] g = 0 for x in TICKETS: prompt = prefix + f"Talep {x['id']} : {x['text']}" topic = call(prompt, x, seed) by_group[x["group"]][0] += topic == x["topic"] by_group[x["group"]][1] += 1 by_spelling[x["spelling"]][0] += topic == x["topic"] by_spelling[x["spelling"]][1] += 1 g += len(tokenize(prompt)) m, ru = by_group["merkez"][0] / by_group["merkez"][1], by_group["kirsal"][0] / by_group["kirsal"][1] return m, ru, m - ru, [a / b for a, b in by_spelling], g print("setup central rural gap gap B localA localB input") for name, prefix in SETUPS: m, ru, a, y, g = measure(prefix) ab = measure(prefix, SEED + 1)[2] print(f"{name:<18} {m:>7.4f} {ru:>7.4f} {a:>8.4f} {ab:>10.4f} {y[1]:>7.4f} " f"{y[2]:>7.4f} {g:>6}") t, e = measure("")[2], measure(NOTICE + EXAMPLES)[2] print(f"\nbaseline gap {t:.4f}, closed by the prompt {t - e:.4f} " f"({(t - e) / t:.4f} share), remaining {e:.4f}") ``` ``` setup central rural gap gap B localA localB input baseline 0.9016 0.5513 0.3504 0.3504 0.4848 0.3023 2624 + notice 0.9098 0.5513 0.3586 0.3457 0.4848 0.3256 6424 + balanced example 0.9426 0.6538 0.2888 0.2888 0.8788 0.3023 17024 + both 0.9426 0.6538 0.2888 0.2760 0.8788 0.3023 20824 baseline gap 0.3504, closed by the prompt 0.0616 (0.1758 share), remaining 0.2888 ``` The second row best fits intuition and sits emptiest in the table. Writing "her yazim esit olcutle degerlendirilir" (every spelling is judged by an equal standard) into the prompt leaves the rural hit rate **at 0.5513** and **raises** the gap **to 0.3586**; with a two-run difference of 0.0129 for the row, even this worsening of 0.0082 counts as unmeasured. The only thing measured is the cost: input tokens **from 2624 to 6424**. A declaration of intent adds no cue to the model. The third row carries a real gain: the balanced example brings the gap down to **0.2888**, a gain of **0.0616** against a two-run difference of **0.0000**. Its cost is **14400 tokens**, 6.5 times the baseline. That the examples also raise the central hit rate (from 0.9016 to 0.9426) is why the gap closes by less than the rural gain. The last two columns explain the remainder. The hit rate of the sampled **local A** spelling rises **from 0.4848 to 0.8788**; the unsampled **local B** spelling does not move from **0.3023**. **A prompt does not solve data bias.** Only a **0.1758 share** of the baseline gap closed; the rest closes not with a better sentence but by building the cue table from local spellings too. #### Summary - A subgroup gap comes not from the model recognizing the group but from **which texts the cue table was built from**; at baseline central is 0.9016, rural is 0.5513 and the gap is 0.3504. - The bias-reducing notice moves the gap to 0.3586 and adds 3800 tokens; with a two-run difference of 0.0129, this change counts as unmeasured. - The balanced example brings the gap down to 0.2888 and adds 14400 tokens; the 0.0616 gain is measured, but the examples also raise the central hit rate, so the gap closes by less than the rural gain. - The hit rate of the sampled local spelling rises from 0.4848 to 0.8788 and the unsampled spelling stays fixed at 0.3023: the prompt closes a 0.1758 share of the gap, and the remaining 0.2888 is data bias. #### Course Wrap-Up The accounting of twenty-two lessons is in a single table, and every number was read from its own lesson. The rows share no unit; the second and third columns state **by which measure** each row is read. **The difference uses a single sign convention: the changed prompt's number minus the previous prompt's number**, in parentheses in the third column. | Lesson | Previous prompt | Changed prompt | Cost | |---|---|---|---| | Design 01 What Is a Prompt | context-free prompt 0.7083 | full prompt 0.7833 (+0.0750) | +1797 tokens; two-run difference 0.2000 → 0.0583 | | Design 02 System Instruction | prompt without instruction 0.4583 | medium instruction 0.7500 (+0.2917) | +2871 tokens; long instruction +3487 tokens (−0.0083) | | Design 03 Specifying Role and Behavior | baseline prompt 0.7500 | role prefix 0.6833 (−0.0667) | +1080 to +3240 tokens; seed band 0.1500, all four prefixes unmeasurable | | Design 04 Clear and Measurable Requests | ambiguous request, 4.81 distinct responses | closed set 3.89 (−0.92) | +15 tokens per request; unlabeled share 0.4333 → 0.2889 | | Design 05 Format and Length Control | free response, parseable 0.0000 | format notice 0.9750 (+0.9750) | input 6140 → 8780, output 1392 → 2026 | | Design 06 Using Examples | zero examples 0.8571 | three examples 0.9643 (+0.1071) | +4928 tokens, +44 per request | | Design 07 Stating Constraints and Prohibitions | prompt without notice, violation 0.2583 | positive notice 0.0167 (−0.2416) | +3360 tokens, 115.9 per prevented violation; a negative constraint makes the violation 0.5333 | | Design 08 Prompt Iteration | starting prompt, test 0.3778 | fifth iteration 0.9333 (+0.5555) | 2640 calls; falls to 0.8111 at the eighth | | Reasoning 01 Chain of Thought | direct answer, calculation 0.3000 | intermediate step prompt 0.8833 (+0.5833) | tokens 1.82 times; unit cost 243.2 → 150.1 | | Reasoning 02 Self-Consistency | single sample 0.7667 | nine samples 0.8833 (+0.1167) | calls 60 → 540, tokens 7954 → 71211, per correct answer 172.9 → 1343.6 | | Reasoning 03 Tree of Thoughts | solution in a single prompt 0.4792 | branch 3 tree 0.6250 (+0.1458) | calls 48 → 795, tokens 43182; branch 4 (−0.0208) | | Reasoning 04 Step-Back Prompting | single-stage prompt 0.5417 | two-stage prompt 0.7500 (+0.2083) | calls 48 → 96, tokens 1.66 times, per correct code 198 → 237 | | Reasoning 05 Reason and Act | action space prompt, valid field 0.1000 | prompt with shortened thought 0.8000 (+0.7000) | input 5300 → 8060; per valid field 645.8 → 102.1 tokens | | Reasoning 06 Reasoning Models | single-pass class, two-field task | reasoning class (+0.1000 and +0.1667) | tokens 1.51 times, per correct answer 83.9 → 110.1 | | Reasoning 07 Prompt Ensemble | best single prompt 0.7833 | majority vote of three prompts 0.8000 (+0.0167) | calls 120 → 360, per correct label 61.1 → 180.7 tokens | | Reasoning 08 Automatic Prompt Optimization | single candidate, test 0.6750 | selected from ten candidates 0.6500 (−0.0250) | calls 40 → 400; validation +0.1250, optimism share 0.2500 | | Output 01 Sampling Parameters | baseline setting 0.6417 | all three narrowed together 0.8667 (+0.2250) | input 5300 fixed, output 191 → 204, calls 120 → 120; run difference 0.5667 → 0.0083 | | Output 02 Penalties and Stop Sequences | plain prompt, repetition 0.3431 | penalty and stop sequence 0.0578 (−0.2853) | output 4073 → 2653, input fixed; usability 0.8833 → 0.8333 | | Output 03 Structured Output | format description, conformance 0.8000 | imposed schema 1.0000 (+0.2000) | input 5614 → 2974 (−2640 tokens), output 1789 → 1810 | | Output 04 Output Validation | schema and business rule, acceptance 0.6000 | one retry 0.8167 (+0.2167) | +48 calls, +2947 tokens, 1.85 calls per recovery | | Output 05 Prompt and Context Engineering Difference | plain prompt, both correct 0.4083 | rule and code table 0.8667 (+0.4584) | input 1534 → 17734; instruction fixes 10 errors with 1680 tokens, information fixes 66 with 14520 | | Output 06 Bias-Reducing Prompts | plain prompt, gap 0.3504 | notice and balanced example 0.2888 (−0.0616) | input 2624 → 20824; unsampled spelling fixed at 0.3023 | **This is the rule of the course: a prompt is a design object; a prompt changed without measurement is not an improvement but a guess.** In no row did the model change: the same simulator, seed and set of fictional tickets. Only the text changed, and the fourth column states twenty-two times that every change of it has a cost. The second claim of the course also reads from the same table, by name: **reasoning techniques are expensive and do not pay off on every task.** In Reasoning 01 chain of thought raised tokens 1.81 times in classification and left the difference inside the band; in Reasoning 04 the step-back prompt raised them 2.69 times on the grade task without moving the hit rate; in Reasoning 06 the reasoning class raised them 1.75 times on the single-field task and left the difference inside the band of 0.1167; in Reasoning 07 the prompt ensemble tripled calls and gained +0.0167; in Reasoning 08 the search budget grew tenfold and the test fell by 0.0250. The same reading holds outside reasoning: in Design 02 the long instruction added 3487 tokens and lost 0.0083, in Design 06 going from the third to the eighth example gained 0.0089 for 8736 tokens, in Output 03 the example-bearing description produced a difference equal to the two-run difference for 2040 tokens, and in Output 04 the second retry stays below that difference at 1366 tokens. The third number stands alone: **in three of the twenty-two lessons**, the difference in the table stayed below the two-run difference of the same lesson — **Design 03** (role specification), **Reasoning 07** (prompt ensemble) and **Reasoning 08** (automatic prompt optimization). In all three the cost column is full and the gain column empty. Over twenty-two lessons the prompt became a design object and the cost of every change was written down. But one assumption was never questioned: that every call happens **alone, immediately and without error**. The number of calls was counted as a cost item; what happens to these prompts where calls travel over a network, queue up, are cut off midway, hit a rate limit and **spend tokens instead of money** was not asked. A prompt ensemble of three hundred sixty calls and a prompt search of two thousand six hundred forty calls were measured in a sequential loop on a single machine. The next course, Building Applications with Model APIs, opens this assumption.