--- title: 'Sampling Parameters' source: 'https://academia.sh/en/courses/prompt-engineering/sampling-parameters' course: 'Prompt Engineering' language: en updated: '2026-10-21T05:02:33+00:00' license: 'CC BY-SA 4.0' objectives: - 'Shows what temperature, top-p (nucleus) and top-k each do to the distribution separately.' - 'Measures nine combinations of settings on the same prompt with two runs and counts which of them narrows the two-run difference.' - 'Compares how the same settings lower diversity in a free-form response task.' - 'Calculates the token and call cost of a settings change.' --- # Sampling Parameters Measuring temperature, top-p (nucleus) and top-k together on the same prompt: label accuracy 0.6417 and two-run difference 0.5667 on the base setting, 0.8750 and 0.0417 at temperature 0.2, 0.8417 and 0.1333 at top-p 0.50, 0.7833 and 0.3583 at top-k 3, 0.8667 and 0.0083 when all three narrow together. The fixed cut leaves three candidates on average while the top-p cut leaves 1.20 candidates. Input tokens stay constant at 5300 and the call count stops at 120, so the cost of the change is thirteen output tokens. The previous sixteen lessons treated the text of the prompt as a design object: the instruction was arranged, examples were placed, constraints were written, the model was asked to reason step by step. In all of these lessons what was measured was **what was written into the prompt**. The response itself always came back as free text, and how it was produced was never asked. This topic opens from there. A response is drawn from a distribution, and that draw has three settings: the distribution temperature, top-p (nucleus) and top-k. These three settings were defined in the Introduction to AI Engineering course; their definitions are not repeated here. The question of this lesson is different: **what happens when all three are used at the same time**, which one helps on which task and which one narrows the **difference between two runs**. - **OC1.** The code is a **simulator** and does not call a real model endpoint. The support tickets, the topics and the document vocabulary are **fictional**; seed **20260218**. - **OC2.** The previous prompt is fixed: an instruction that lists the label set and asks for a single label. This prompt **never changes** during the whole lesson; the only thing that changes is the sampling setting of the call. - **OC3.** The criterion is twofold. In the labeling task **label accuracy** (higher is better) is measured, in the free-form response task the **number of distinct responses** (diversity). - **OC4.** The **two-run difference** is the **share of records whose decision changes** between seed 20260218 and seed 20260219 on the same 120 tickets. A difference that a change adds counts as unmeasured if it is smaller than this number. - **OC5.** Temperature reshapes the distribution; it **does not shorten the candidate list**. Top-p (nucleus) and top-k, on the other hand, **cut the list**. - **OC6.** The number of candidates is printed as well: the number of candidates left by the fixed cut and the number left by the adaptive cut appear in the same table. - **OC7.** The simulator's internal cue table has **misleading ties**; no sampling setting corrects them. - **OC8.** The cost is written in **tokens**: input tokens, output tokens and the number of calls. ## The Separate Jobs of the Three Settings The three settings work in the same place: after the scores are turned into a probability distribution and right before the draw. Temperature is a divisor and sharpens or flattens the distribution. Top-k cuts the list **at a fixed number** from the start. Top-p (nucleus) looks not at a number but at the **sum of probabilities** and takes candidates until it reaches the threshold. ```python # SIMULATOR -- FICTIONAL. No real model endpoint is called; the code below is a language # model simulator written with the standard library. The tickets are FICTIONAL too. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUT = 240, 24 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): # FICTIONAL tokenizer, a piece has at most five characters out = [] for piece in text.lower().replace(",", " , ").split(): while len(piece) > 5: out.append(piece[:5]) piece = piece[5:] out.append(piece) return out def trace_id(text): # depends on the prompt, independent of the run h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"), ("faturami odedim ama sistemde borc gorunuyor", "fatura"), ("fatura ayrintisini nereden gorebilirim", "fatura"), ("evde su akmiyor bir ariza var galiba", "ariza"), ("baglanti bozuk oldu ve hic calismiyor", "ariza"), ("ariza kaydi actim ama ekip gelmedi", "ariza"), ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"), ("sayac degisimi icin randevu istiyorum", "sayac"), ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"), ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"), ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"), ("birim ucret abone tipine gore degisiyor mu", "tarife"), ("mahallede kesinti var ne zaman gelecek", "kesinti"), ("planli kesinti duyurusu nereden takip edilir", "kesinti"), ("aksamdan beri kesinti yasiyoruz", "kesinti")] EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim", "acil bilgi almam gerekiyor", "kayit numaram elimde yok"] TICKETS = [] for i in range(120): # 120 FICTIONAL support tickets r = rng(SEED + i) topic = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] text, _ = [x for x in PATTERNS if x[1] == topic][int(r() * 3)] TICKETS.append({"id": 4001 + i, "topic": topic, "text": text + (" " + EXTRAS[int(r() * 4)] if r() < 0.34 else "")}) CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "borc": "odeme", "odedi": "odeme", "ucret": "odeme", "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet", "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"} TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} # misleading cues FREE_FORM = ["talebiniz {k} konusuyla ilgili gorunuyor", "bu kayit {k} basligina girer , ilgili ekibe yonlendirildi", "kaydiniz alindi , konuyu inceleyip donus yapacagiz"] COUNTER = [0] def distribution(toks, temperature, top_p, top_k): """The three settings work here and only here: temperature reshapes the distribution, top-k leaves a fixed number of candidates, top-p (nucleus) looks at the probability sum.""" cands = [k for k in CONCEPTS if k[:5] in toks] p = {k: 0.15 for k in (cands if len(cands) >= 3 else CONCEPTS)} for t in toks: for table, w in ((CUES, 1.0), (TRAPS, 1.3)): if table.get(t) in p: p[table[t]] += w names = sorted(p, key=lambda a: (-p[a], a)) if temperature <= 0.0: return [names[0]], [1.0] # most probable choice e = [math.exp(p[a] / temperature) for a in names] probs = [v / sum(e) for v in e] names, probs = names[:top_k], probs[:top_k] # top-k selection n, mass = 0, 0.0 while mass < top_p and n < len(probs): # top-p (nucleus) sampling mass += probs[n] n += 1 names, probs = names[:n], probs[:n] return names, [v / sum(probs) for v in probs] def call(prompt, seed=SEED, temperature=1.0, top_p=1.0, top_k=9, max_out=MAX_OUT): COUNTER[0] += 1 toks = tokenize(prompt) if len(toks) > WINDOW: return {"status": "context_overflow", "input": len(toks), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: return {"status": "rate_limit", "input": len(toks), "output": 0, "response": ""} r = rng(seed ^ trace_id(prompt)) names, probs = distribution(toks, temperature, top_p, top_k) k = names[pick(r(), probs)] y = k if "tek" in toks else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k) out = tokenize(y) if len(out) > max_out: # stop condition out, y = out[:max_out], " ".join(out[:max_out]) return {"status": "ok", "input": len(toks), "output": len(out), "response": y} def read_label(y): for s in y.lower().split(): if s in TOPICS: return s return None LABEL_PROMPT = ("Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife , " "kesinti . Yanit olarak tek etiket yaz . Talep: ") b = tokenize(LABEL_PROMPT + TICKETS[6]["text"]) print(f"FICTIONAL ticket set {len(TICKETS)} records; sample ticket {TICKETS[6]['id']}") print(f"prompt {len(b)} tokens, actual topic {TICKETS[6]['topic']}") print("setting candidates top three probabilities") for name, t, p, k in (("base 1.0 / 1.00 / 9", 1.0, 1.0, 9), ("temperature 0.2", 0.2, 1.0, 9), ("top-p 0.50", 1.0, 0.5, 9), ("top-k 3", 1.0, 1.0, 3)): a, o = distribution(b, t, p, k) print(f"{name:<24} {len(a):>10} {' '.join(f'{v:.3f}' for v in o[:3])}") ``` ``` FICTIONAL ticket set 120 records; sample ticket 4007 prompt 42 tokens, actual topic sayac setting candidates top three probabilities base 1.0 / 1.00 / 9 5 0.564 0.207 0.076 temperature 0.2 5 0.993 0.007 0.000 top-p 0.50 1 1.000 top-k 3 3 0.665 0.245 0.090 ``` The four rows show three separate jobs. At temperature 0.2 the number of candidates **stays at five** but the highest probability goes from 0.564 to 0.993: the list did not shorten, the distribution sharpened. Top-p 0.50 returns with a single candidate, because the probability of the first candidate is already 0.564 and exceeds the threshold on its own. Top-k 3 brings the list down to three, but when the probabilities of the remaining three candidates are rescaled, the second candidate **rises** from 0.207 to 0.245. Cutting strengthens what remains. The distinction that comes out of this carries the rest of the lesson: temperature is a **reshaping**, top-p (nucleus) is an **adaptive cut**, top-k is a **fixed cut**. ## The Three Settings Together Nine combinations of settings are measured on the same prompt, the same 120 tickets and the same simulator. In the labeling task the accuracy is printed, in the free-form response task the number of distinct responses. The two-run difference is the share of records whose decision changes in the second run, taken with the seed changed. ```python FREE_PROMPT = "Su destek talebine kisa bir yanit yaz . Talep: " SETTINGS = [("base 1.0 / 1.00 / 9", 1.0, 1.0, 9), ("temperature 0.6", 0.6, 1.0, 9), ("temperature 0.2", 0.2, 1.0, 9), ("most probable choice", 0.0, 1.0, 9), ("top-p 0.50", 1.0, 0.5, 9), ("top-p 0.20", 1.0, 0.2, 9), ("top-k 3", 1.0, 1.0, 3), ("top-k 1", 1.0, 1.0, 1), ("all three together", 0.6, 0.5, 3)] def run(instruction, t, p, k, seed): COUNTER[0] = 0 return [read_label(call(instruction + x["text"], seed=seed, temperature=t, top_p=p, top_k=k)["response"]) for x in TICKETS] print("setting accuracy A accuracy B two-run difference distinct responses candidates") for name, t, p, k in SETTINGS: a, b = run(LABEL_PROMPT, t, p, k, SEED), run(LABEL_PROMPT, t, p, k, SEED + 1) diff = sum(1 for x, y in zip(a, b) if x != y) / len(a) acc_a = sum(x == tk["topic"] for x, tk in zip(a, TICKETS)) / len(a) acc_b = sum(x == tk["topic"] for x, tk in zip(b, TICKETS)) / len(b) COUNTER[0] = 0 distinct = len({call(FREE_PROMPT + x["text"], temperature=t, top_p=p, top_k=k)["response"] for x in TICKETS}) cands = sum(len(distribution(tokenize(LABEL_PROMPT + tk["text"]), t, p, k)[0]) for tk in TICKETS) / len(TICKETS) print(f"{name:<22} {acc_a:>10.4f} {acc_b:>10.4f} {diff:>18.4f} {distinct:>18} {cands:>10.2f}") for name, t, p, k in (SETTINGS[0], SETTINGS[-1]): COUNTER[0] = 0 y = [call(LABEL_PROMPT + x["text"], temperature=t, top_p=p, top_k=k) for x in TICKETS] print(f"cost {name:<22} input {sum(z['input'] for z in y)} " f"output {sum(z['output'] for z in y)} calls {COUNTER[0]}") ``` ``` setting accuracy A accuracy B two-run difference distinct responses candidates base 1.0 / 1.00 / 9 0.6417 0.6000 0.5667 15 5.03 temperature 0.6 0.8167 0.8250 0.2833 17 5.03 temperature 0.2 0.8750 0.8917 0.0417 10 5.03 most probable choice 0.8667 0.8667 0.0000 10 1.00 top-p 0.50 0.8417 0.9667 0.1333 15 1.20 top-p 0.20 0.8667 0.8667 0.0000 10 1.00 top-k 3 0.7833 0.7833 0.3583 15 3.00 top-k 1 0.8667 0.8667 0.0000 10 1.00 all three together 0.8667 0.8667 0.0083 10 1.01 cost base 1.0 / 1.00 / 9 input 5300 output 191 calls 120 cost all three together input 5300 output 204 calls 120 ``` The base row is an unmeasurable row. The accuracy looks like **0.6417**, but the two-run difference is **0.5667**: in more than half of the records the decision changes when the seed changes. No prompt improvement measured on this base can be defended, because the difference sought stays below the noise. All three settings narrow this difference, but not to the same extent. Temperature 0.2 brings the difference down to **0.0417** and raises the accuracy to **0.8750**, the highest in the table. What stands out is that this is higher than the most probable choice (**0.8667**) too: sharpening the distribution gave a better result than cutting it off entirely. Top-p 0.50 brings the difference down to **0.1333** while keeping the mean number of candidates at **1.20**; that is, it drops to a single candidate on most tickets and leaves the second candidate open on the few tickets where the evidence is weak. This is the **adaptive** cut. The top-k 3 row is the most instructive row in the table. The number of candidates is fixed at **3.00**, and the two-run difference of **0.3583** is the largest value apart from the base. A fixed cut leaves two rival candidates unnecessarily on the tickets where the evidence is strong, and temperature 1.0 keeps making them compete. Cutting alone does not close the uncertainty; **what the cut is made by** decides. The free-form response column shows the opposite direction. Narrowing gains in labeling and loses here: the number of distinct responses is **17** at temperature 0.6 and falls to **10** in the most narrowed settings; it stays at **15** with top-p 0.50 and top-k 3. The same three settings work in two directions on two tasks. In a task with a single correct answer, uncertainty is a defect; in a task where a separate sentence is expected for each subscriber, uncertainty is **the product itself**. Ten distinct responses remain because the settings do not affect at all the draw that selects the **format** of the response: a setting does not close every source of uncertainty. ## A Change Whose Cost Is Zero Tokens The cost line of this lesson differs from the rest of the course. The previous prompt and the changed prompt are **the exact same text**: the input tokens are **5300** in both settings, the number of calls is **120** in both. The output tokens go from 191 to 204, because the narrowed setting chooses the longer-written labels slightly more often. The total cost is **13 tokens**, that is, 0.11 tokens per request. What is bought in return is measurability itself. On the base setting the two-run difference is 0.5667, while when all three are narrowed together it falls to **0.0083**. In the lessons from here on, a gain of 0.05 added by a prompt change is a meaningful number; on the base setting the same gain could not have been measured. **The rule of the course turns into a precondition here: a prompt changed without being measured is not an improvement but a guess, and measurement is possible only when the two-run difference is small enough.** This also has a limit that must be written down. In the setting where all three are narrowed, the accuracy stays at **0.8667**, and most of the 16 remaining wrong records come from the simulator's misleading ties. Sampling settings close the uncertainty; they **do not close systematic error**. A response that is stably wrong is stable too. ## Summary - Temperature shapes the distribution and does not shorten the candidate list; top-p (nucleus) makes an **adaptive** cut by looking at the probability sum; top-k leaves a **fixed** number of candidates. - On the base setting the label accuracy is 0.6417 but the two-run difference is 0.5667; on this base no prompt change can be measured. - Temperature 0.2 brings the difference down to 0.0417 while raising the accuracy to 0.8750, the highest, and passes the 0.8667 of the most probable choice; sharpening the distribution gave a better result than cutting it off entirely. - Top-k 3 fixes the number of candidates at 3.00 and leaves the two-run difference at 0.3583; top-p 0.50 brings the difference down to 0.1333 with 1.20 candidates. - The same settings lower the diversity in the free-form response task from 17 to 10: narrowing is a gain in one task and a loss in the other. - The cost of the change is 13 output tokens and zero extra calls; the input tokens stay constant at 5300. ## Next Step The sampling settings controlled which concept would be chosen, but never controlled **where the response would end**. In this lesson the output length was cut by a single hard limit, and how many responses were cut was not counted. In the same way, a response repeating the same sentence within itself was never measured either: sharpening the distribution makes the most probable phrase even more probable, and in long responses this turns into self-repetition. The next lesson adds two new controls and measures both with numbers: how much the repetition penalty lowers the repetition rate, how many responses the stop sequence cuts and how many of them it cuts **prematurely**.