Sampling Parameters

The previous sixteen lessons treated the text of the prompt as a design object: the instruction was arranged, examples were placed, constraints were written, the model was asked to reason step by step. In all of these lessons what was measured was what was written into the prompt. The response itself always came back as free text, and how it was produced was never asked.

This topic opens from there. A response is drawn from a distribution, and that draw has three settings: the distribution temperature, top-p (nucleus) and top-k. These three settings were defined in the Introduction to AI Engineering course; their definitions are not repeated here. The question of this lesson is different: what happens when all three are used at the same time, which one helps on which task and which one narrows the difference between two runs.

• OC1. The code is a simulator and does not call a real model endpoint. The support tickets, the topics and the document vocabulary are fictional; seed 20260218.
• OC2. The previous prompt is fixed: an instruction that lists the label set and asks for a single label. This prompt never changes during the whole lesson; the only thing that changes is the sampling setting of the call.
• OC3. The criterion is twofold. In the labeling task label accuracy (higher is better) is measured, in the free-form response task the number of distinct responses (diversity).
• OC4. The two-run difference is the share of records whose decision changes between seed 20260218 and seed 20260219 on the same 120 tickets. A difference that a change adds counts as unmeasured if it is smaller than this number.
• OC5. Temperature reshapes the distribution; it does not shorten the candidate list. Top-p (nucleus) and top-k, on the other hand, cut the list.
• OC6. The number of candidates is printed as well: the number of candidates left by the fixed cut and the number left by the adaptive cut appear in the same table.
• OC7. The simulator’s internal cue table has misleading ties; no sampling setting corrects them.
• OC8. The cost is written in tokens: input tokens, output tokens and the number of calls.

The Separate Jobs of the Three Settings

The three settings work in the same place: after the scores are turned into a probability distribution and right before the draw. Temperature is a divisor and sharpens or flattens the distribution. Top-k cuts the list at a fixed number from the start. Top-p (nucleus) looks not at a number but at the sum of probabilities and takes candidates until it reaches the threshold.

# SIMULATOR -- FICTIONAL. No real model endpoint is called; the code below is a language
# model simulator written with the standard library. The tickets are FICTIONAL too.
import math

SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW, MAX_OUT = 240, 24


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(text):                     # FICTIONAL tokenizer, a piece has at most five characters
    out = []
    for piece in text.lower().replace(",", " , ").split():
        while len(piece) > 5:
            out.append(piece[:5])
            piece = piece[5:]
        out.append(piece)
    return out


def trace_id(text):                     # depends on the prompt, independent of the run
    h = 2166136261
    for c in text:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"),
            ("faturami odedim ama sistemde borc gorunuyor", "fatura"),
            ("fatura ayrintisini nereden gorebilirim", "fatura"),
            ("evde su akmiyor bir ariza var galiba", "ariza"),
            ("baglanti bozuk oldu ve hic calismiyor", "ariza"),
            ("ariza kaydi actim ama ekip gelmedi", "ariza"),
            ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"),
            ("sayac degisimi icin randevu istiyorum", "sayac"),
            ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"),
            ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"),
            ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"),
            ("birim ucret abone tipine gore degisiyor mu", "tarife"),
            ("mahallede kesinti var ne zaman gelecek", "kesinti"),
            ("planli kesinti duyurusu nereden takip edilir", "kesinti"),
            ("aksamdan beri kesinti yasiyoruz", "kesinti")]
EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim",
          "acil bilgi almam gerekiyor", "kayit numaram elimde yok"]
TICKETS = []
for i in range(120):                    # 120 FICTIONAL support tickets
    r = rng(SEED + i)
    topic = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    text, _ = [x for x in PATTERNS if x[1] == topic][int(r() * 3)]
    TICKETS.append({"id": 4001 + i, "topic": topic,
                    "text": text + (" " + EXTRAS[int(r() * 4)] if r() < 0.34 else "")})

CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"]
CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza",
        "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac",
        "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife",
        "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti",
        "mahal": "kesinti", "borc": "odeme", "odedi": "odeme", "ucret": "odeme",
        "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet", "bilgi": "bilgi",
        "ogren": "bilgi", "nered": "bilgi"}
TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura",
         "ekip": "kesinti", "takip": "tarife", "giril": "fatura"}   # misleading cues
FREE_FORM = ["talebiniz {k} konusuyla ilgili gorunuyor",
             "bu kayit {k} basligina girer , ilgili ekibe yonlendirildi",
             "kaydiniz alindi , konuyu inceleyip donus yapacagiz"]
COUNTER = [0]


def distribution(toks, temperature, top_p, top_k):
    """The three settings work here and only here: temperature reshapes the distribution,
    top-k leaves a fixed number of candidates, top-p (nucleus) looks at the probability sum."""
    cands = [k for k in CONCEPTS if k[:5] in toks]
    p = {k: 0.15 for k in (cands if len(cands) >= 3 else CONCEPTS)}
    for t in toks:
        for table, w in ((CUES, 1.0), (TRAPS, 1.3)):
            if table.get(t) in p:
                p[table[t]] += w
    names = sorted(p, key=lambda a: (-p[a], a))
    if temperature <= 0.0:
        return [names[0]], [1.0]                    # most probable choice
    e = [math.exp(p[a] / temperature) for a in names]
    probs = [v / sum(e) for v in e]
    names, probs = names[:top_k], probs[:top_k]     # top-k selection
    n, mass = 0, 0.0
    while mass < top_p and n < len(probs):          # top-p (nucleus) sampling
        mass += probs[n]
        n += 1
    names, probs = names[:n], probs[:n]
    return names, [v / sum(probs) for v in probs]


def call(prompt, seed=SEED, temperature=1.0, top_p=1.0, top_k=9, max_out=MAX_OUT):
    COUNTER[0] += 1
    toks = tokenize(prompt)
    if len(toks) > WINDOW:
        return {"status": "context_overflow", "input": len(toks), "output": 0, "response": ""}
    if COUNTER[0] % 37 == 0:
        return {"status": "rate_limit", "input": len(toks), "output": 0, "response": ""}
    r = rng(seed ^ trace_id(prompt))
    names, probs = distribution(toks, temperature, top_p, top_k)
    k = names[pick(r(), probs)]
    y = k if "tek" in toks else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k)
    out = tokenize(y)
    if len(out) > max_out:                          # stop condition
        out, y = out[:max_out], " ".join(out[:max_out])
    return {"status": "ok", "input": len(toks), "output": len(out), "response": y}


def read_label(y):
    for s in y.lower().split():
        if s in TOPICS:
            return s
    return None


LABEL_PROMPT = ("Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife , "
                "kesinti . Yanit olarak tek etiket yaz . Talep: ")
b = tokenize(LABEL_PROMPT + TICKETS[6]["text"])
print(f"FICTIONAL ticket set {len(TICKETS)} records; sample ticket {TICKETS[6]['id']}")
print(f"prompt {len(b)} tokens, actual topic {TICKETS[6]['topic']}")
print("setting                  candidates  top three probabilities")
for name, t, p, k in (("base 1.0 / 1.00 / 9", 1.0, 1.0, 9),
                      ("temperature 0.2", 0.2, 1.0, 9),
                      ("top-p 0.50", 1.0, 0.5, 9),
                      ("top-k 3", 1.0, 1.0, 3)):
    a, o = distribution(b, t, p, k)
    print(f"{name:<24} {len(a):>10}  {' '.join(f'{v:.3f}' for v in o[:3])}")

FICTIONAL ticket set 120 records; sample ticket 4007
prompt 42 tokens, actual topic sayac
setting                  candidates  top three probabilities
base 1.0 / 1.00 / 9               5  0.564 0.207 0.076
temperature 0.2                   5  0.993 0.007 0.000
top-p 0.50                        1  1.000
top-k 3                           3  0.665 0.245 0.090

The four rows show three separate jobs. At temperature 0.2 the number of candidates stays at five but the highest probability goes from 0.564 to 0.993: the list did not shorten, the distribution sharpened. Top-p 0.50 returns with a single candidate, because the probability of the first candidate is already 0.564 and exceeds the threshold on its own. Top-k 3 brings the list down to three, but when the probabilities of the remaining three candidates are rescaled, the second candidate rises from 0.207 to 0.245. Cutting strengthens what remains.

The distinction that comes out of this carries the rest of the lesson: temperature is a reshaping, top-p (nucleus) is an adaptive cut, top-k is a fixed cut.

The Three Settings Together

Nine combinations of settings are measured on the same prompt, the same 120 tickets and the same simulator. In the labeling task the accuracy is printed, in the free-form response task the number of distinct responses. The two-run difference is the share of records whose decision changes in the second run, taken with the seed changed.

FREE_PROMPT = "Su destek talebine kisa bir yanit yaz . Talep: "
SETTINGS = [("base 1.0 / 1.00 / 9", 1.0, 1.0, 9), ("temperature 0.6", 0.6, 1.0, 9),
            ("temperature 0.2", 0.2, 1.0, 9), ("most probable choice", 0.0, 1.0, 9),
            ("top-p 0.50", 1.0, 0.5, 9), ("top-p 0.20", 1.0, 0.2, 9),
            ("top-k 3", 1.0, 1.0, 3), ("top-k 1", 1.0, 1.0, 1),
            ("all three together", 0.6, 0.5, 3)]


def run(instruction, t, p, k, seed):
    COUNTER[0] = 0
    return [read_label(call(instruction + x["text"], seed=seed, temperature=t,
                            top_p=p, top_k=k)["response"]) for x in TICKETS]


print("setting                accuracy A  accuracy B  two-run difference  distinct responses  candidates")
for name, t, p, k in SETTINGS:
    a, b = run(LABEL_PROMPT, t, p, k, SEED), run(LABEL_PROMPT, t, p, k, SEED + 1)
    diff = sum(1 for x, y in zip(a, b) if x != y) / len(a)
    acc_a = sum(x == tk["topic"] for x, tk in zip(a, TICKETS)) / len(a)
    acc_b = sum(x == tk["topic"] for x, tk in zip(b, TICKETS)) / len(b)
    COUNTER[0] = 0
    distinct = len({call(FREE_PROMPT + x["text"], temperature=t, top_p=p,
                         top_k=k)["response"] for x in TICKETS})
    cands = sum(len(distribution(tokenize(LABEL_PROMPT + tk["text"]), t, p, k)[0])
                for tk in TICKETS) / len(TICKETS)
    print(f"{name:<22} {acc_a:>10.4f}  {acc_b:>10.4f}  {diff:>18.4f}  {distinct:>18}  {cands:>10.2f}")
for name, t, p, k in (SETTINGS[0], SETTINGS[-1]):
    COUNTER[0] = 0
    y = [call(LABEL_PROMPT + x["text"], temperature=t, top_p=p, top_k=k) for x in TICKETS]
    print(f"cost {name:<22} input {sum(z['input'] for z in y)} "
          f"output {sum(z['output'] for z in y)} calls {COUNTER[0]}")

setting                accuracy A  accuracy B  two-run difference  distinct responses  candidates
base 1.0 / 1.00 / 9        0.6417      0.6000              0.5667                  15        5.03
temperature 0.6            0.8167      0.8250              0.2833                  17        5.03
temperature 0.2            0.8750      0.8917              0.0417                  10        5.03
most probable choice       0.8667      0.8667              0.0000                  10        1.00
top-p 0.50                 0.8417      0.9667              0.1333                  15        1.20
top-p 0.20                 0.8667      0.8667              0.0000                  10        1.00
top-k 3                    0.7833      0.7833              0.3583                  15        3.00
top-k 1                    0.8667      0.8667              0.0000                  10        1.00
all three together         0.8667      0.8667              0.0083                  10        1.01
cost base 1.0 / 1.00 / 9    input 5300 output 191 calls 120
cost all three together     input 5300 output 204 calls 120

The base row is an unmeasurable row. The accuracy looks like 0.6417, but the two-run difference is 0.5667: in more than half of the records the decision changes when the seed changes. No prompt improvement measured on this base can be defended, because the difference sought stays below the noise.

All three settings narrow this difference, but not to the same extent. Temperature 0.2 brings the difference down to 0.0417 and raises the accuracy to 0.8750, the highest in the table. What stands out is that this is higher than the most probable choice (0.8667) too: sharpening the distribution gave a better result than cutting it off entirely. Top-p 0.50 brings the difference down to 0.1333 while keeping the mean number of candidates at 1.20; that is, it drops to a single candidate on most tickets and leaves the second candidate open on the few tickets where the evidence is weak. This is the adaptive cut.

The top-k 3 row is the most instructive row in the table. The number of candidates is fixed at 3.00, and the two-run difference of 0.3583 is the largest value apart from the base. A fixed cut leaves two rival candidates unnecessarily on the tickets where the evidence is strong, and temperature 1.0 keeps making them compete. Cutting alone does not close the uncertainty; what the cut is made by decides.

The free-form response column shows the opposite direction. Narrowing gains in labeling and loses here: the number of distinct responses is 17 at temperature 0.6 and falls to 10 in the most narrowed settings; it stays at 15 with top-p 0.50 and top-k 3. The same three settings work in two directions on two tasks. In a task with a single correct answer, uncertainty is a defect; in a task where a separate sentence is expected for each subscriber, uncertainty is the product itself. Ten distinct responses remain because the settings do not affect at all the draw that selects the format of the response: a setting does not close every source of uncertainty.

A Change Whose Cost Is Zero Tokens

The cost line of this lesson differs from the rest of the course. The previous prompt and the changed prompt are the exact same text: the input tokens are 5300 in both settings, the number of calls is 120 in both. The output tokens go from 191 to 204, because the narrowed setting chooses the longer-written labels slightly more often. The total cost is 13 tokens, that is, 0.11 tokens per request.

What is bought in return is measurability itself. On the base setting the two-run difference is 0.5667, while when all three are narrowed together it falls to 0.0083. In the lessons from here on, a gain of 0.05 added by a prompt change is a meaningful number; on the base setting the same gain could not have been measured. The rule of the course turns into a precondition here: a prompt changed without being measured is not an improvement but a guess, and measurement is possible only when the two-run difference is small enough.

This also has a limit that must be written down. In the setting where all three are narrowed, the accuracy stays at 0.8667, and most of the 16 remaining wrong records come from the simulator’s misleading ties. Sampling settings close the uncertainty; they do not close systematic error. A response that is stably wrong is stable too.

Summary

• Temperature shapes the distribution and does not shorten the candidate list; top-p (nucleus) makes an adaptive cut by looking at the probability sum; top-k leaves a fixed number of candidates.
• On the base setting the label accuracy is 0.6417 but the two-run difference is 0.5667; on this base no prompt change can be measured.
• Temperature 0.2 brings the difference down to 0.0417 while raising the accuracy to 0.8750, the highest, and passes the 0.8667 of the most probable choice; sharpening the distribution gave a better result than cutting it off entirely.
• Top-k 3 fixes the number of candidates at 3.00 and leaves the two-run difference at 0.3583; top-p 0.50 brings the difference down to 0.1333 with 1.20 candidates.
• The same settings lower the diversity in the free-form response task from 17 to 10: narrowing is a gain in one task and a loss in the other.
• The cost of the change is 13 output tokens and zero extra calls; the input tokens stay constant at 5300.

Next Step

The sampling settings controlled which concept would be chosen, but never controlled where the response would end. In this lesson the output length was cut by a single hard limit, and how many responses were cut was not counted. In the same way, a response repeating the same sentence within itself was never measured either: sharpening the distribution makes the most probable phrase even more probable, and in long responses this turns into self-repetition. The next lesson adds two new controls and measures both with numbers: how much the repetition penalty lowers the repetition rate, how many responses the stop sequence cuts and how many of them it cuts prematurely.