Using Examples

The previous lesson wrote the format of the response into the prompt and carried the machine-parseable response share from 0.0000 to 0.9750. What went into the prompt was a recipe: the names, the order and the separator of the fields. The recipe determined the container of the response, but it did not touch the decision itself. Which ticket goes to which label is still something only the model knows, and the prompt contains not a single example of it.

This lesson puts exemplars in place of the recipe. Solved tickets are written into the prompt, and each of them tells the model “this text goes to this label”. Three things will be measured: the hit rate gain per exemplar, the token cost per exemplar, and where the gain stops. At the end of the lesson the measure turns on itself once more: when the same exemplars are written with only their order changed, does the number move?

• PD48. The model is a simulator, its seed is 20260218, and no real endpoint is called. The support tickets and the fault codes are fictional.
• PD49. The format constraint of the previous lesson is held constant. In every run the response is requested as a three-field record and read with the same strict field parser; so the only thing that changes in this lesson is the exemplars.
• PD50. The measurement is made with the most likely choice. With sampling off, the label decision is independent of the run, and the difference that an exemplar adds does not get mixed into the sampling noise; the runs with sampling on are printed separately.
• PD51. The exemplar pool is separate from the measurement set: eight solved tickets are taken into the pool and the remaining 112 tickets are measured. The first five exemplars of the pool cover the five labels once each, and the remaining three bring the second templates.
• PD52. The simulator uses an exemplar like a matching table written into the prompt: every token that the exemplar text shares with the ticket raises the score of the exemplar’s label. The last exemplar weighs more; the order effect comes from here.
• PD53. The cost is written in input tokens. The exemplars are sent again with every request; the input tokens per request grow in direct proportion to the number of exemplars.

Gain and Cost per Example

# SIMULATOR -- FICTIONAL. This code does not call a real model endpoint; it is a
# language model simulator written with the standard library. The tickets are fictional too.
import math

SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW, MAX_OUT = 240, 24


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):                    # FICTIONAL tokenizer: pieces of at most five characters
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def prompt_trace(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
CODES = {"fatura": "ft-21", "ariza": "ar-14", "sayac": "sy-32", "tarife": "tr-05",
         "kesinti": "ks-40", "odeme": "od-11", "sikayet": "sk-09", "bilgi": "bg-02",
         "genel": "gn-00"}
TEMPLATES = {"fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi",
                         "faturami odedim ama sistemde borc gorunuyor",
                         "fatura ayrintisini nereden gorebilirim"],
             "ariza": ["evde su akmiyor bir ariza var galiba",
                       "baglanti bozuk oldu ve hic calismiyor",
                       "ariza kaydi actim ama ekip gelmedi"],
             "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor",
                       "sayac degisimi icin randevu istiyorum",
                       "sayac uzerindeki endeks ile fatura ayni degil"],
             "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum",
                        "tarife degisikligi icin ne yapmam gerekiyor",
                        "birim ucret abone tipine gore degisiyor mu"],
             "kesinti": ["mahallede kesinti var ne zaman gelecek",
                        "planli kesinti duyurusu nereden takip edilir",
                        "aksamdan beri kesinti yasiyoruz"]}
EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim",
          "acil bilgi almam gerekiyor", "kayit numaram elimde yok"]

TICKETS = []
for i in range(120):
    r = rng(SEED + i)
    k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    m = TEMPLATES[k][int(r() * 3)]
    if r() < 0.34:
        m += " " + EXTRAS[int(r() * 4)]
    TICKETS.append({"id": 4001 + i, "text": m, "topic": k})

CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"]
CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura",
        "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza",
        "sayac": "sayac", "endek": "sayac", "okuma": "sayac",
        "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife",
        "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti",
        "borc": "odeme", "odedi": "odeme", "ucret": "odeme",
        "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet",
        "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"}
TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura",
         "ekip": "kesinti", "takip": "tarife", "giril": "fatura"}   # misleading cues
CALL_COUNT = [0]


def parse(prompt):                  # splits the prompt into an example list and the ticket text
    head, _, ticket = prompt.partition("Talep : ")
    examples = []
    for piece in head.split("Ornek : ")[1:]:
        m, _, e = piece.partition(" => ")
        examples.append((m.strip(), e.split(" .")[0].strip()))
    return examples, ticket


def call(prompt, summary, seed=SEED, temperature=0.0, max_out=MAX_OUT):
    CALL_COUNT[0] += 1
    tok = tokenize(prompt)
    empty = {"tokens_in": len(tok), "tokens_out": 0, "response": ""}
    if len(tok) > WINDOW:
        return dict(empty, status="context_overflow")
    if CALL_COUNT[0] % 37 == 0:
        return dict(empty, status="rate_limit")
    r = rng(seed ^ prompt_trace(prompt))
    if len(tok) > 90 and r() < 0.04:            # long prompt , timeout
        return dict(empty, status="timeout")
    examples, ticket = parse(prompt)
    tt = tokenize(ticket)
    cand = [k for k in CONCEPTS if k[:5] in tok]
    if len(cand) < 3:
        cand = CONCEPTS
    p = {k: 0.15 for k in cand}
    for t in tt:                                 # ticket tokens: cue and trap
        for table, w in ((CUES, 1.0), (TRAPS, 1.3)):
            if table.get(t) in p:
                p[table[t]] += w
    for j, (m, e) in enumerate(examples):        # in-context learning
        if e in p:
            w = 0.5 + (j + 1) / len(examples)    # the last example weighs more
            for t in set(tokenize(m)):
                if t in tt:
                    p[e] += 0.30 * w
    if temperature <= 0.0:
        k = max(sorted(p), key=lambda a: p[a])   # most likely choice
    else:
        e = [math.exp(p[a] / temperature) for a in sorted(p)]
        k = sorted(p)[pick(r(), [v / sum(e) for v in e])]
    y = f"etiket : {k} ; kod : {CODES[k]} ; gerekce : {summary}"
    ot = tokenize(y)
    if len(ot) > max_out:
        ot, y = ot[:max_out], " ".join(ot[:max_out])
    return {"status": "ok", "tokens_in": len(tok), "tokens_out": len(ot), "response": y}


def parse_strict(y):                # the strict field parser of the previous lesson
    a = {}
    for part in y.split(";"):
        kv = part.split(":")
        if len(kv) == 2:
            a[kv[0].strip()] = kv[1].strip()
    complete = a.get("etiket") in TOPICS and a.get("kod") == CODES.get(a.get("etiket"))
    return a["etiket"] if complete and a.get("gerekce") else None


INSTRUCTION = ("Talebi su etiketlerden birine ata : fatura , ariza , sayac , tarife , "
               "kesinti . Yanitin bicimi : etiket : <deger> ; kod : <deger> ; "
               "gerekce : <deger> . ")
PICKS = [1, 3, 4, 5, 0, 18, 13, 40]   # first one example per label, then the second template
POOL = [TICKETS[i] for i in PICKS]
DATASET = [t for i, t in enumerate(TICKETS) if i not in PICKS]


def build_prompt(examples, text):
    o = "".join(f"Ornek : {x['text']} => {x['topic']} . " for x in examples)
    return INSTRUCTION + o + "Talep : " + text


def measure(examples, seed=SEED, temperature=0.0):
    CALL_COUNT[0] = 0
    tin = tout = hit = bad = 0
    for x in DATASET:
        s = call(build_prompt(examples, x["text"]), " ".join(x["text"].split()[:3]),
                 seed=seed, temperature=temperature)
        tin, tout = tin + s["tokens_in"], tout + s["tokens_out"]
        bad += s["status"] != "ok"
        hit += parse_strict(s["response"]) == x["topic"]
    return {"hit_rate": hit / len(DATASET), "tokens_in": tin, "tokens_out": tout, "failed": bad}


print(f"FICTIONAL: example pool {len(POOL)} solved tickets , measurement set {len(DATASET)} tickets")
print(f"example format: Ornek : {POOL[0]['text']} => {POOL[0]['topic']} .")
print(f"\n{'examples':>8}  {'label':>7}  {'hit rate':>8}  {'input tokens':>12}"
      f"  {'per request':>11}  {'gain':>7}")
previous = None
for n in range(9):
    d = measure(POOL[:n])
    gain = "-" if previous is None else f"{d['hit_rate'] - previous:+.4f}"
    name = POOL[n - 1]["topic"] if n else "-"
    print(f"{n:>8}  {name:>7}  {d['hit_rate']:>8.4f}  {d['tokens_in']:>12}"
          f"  {d['tokens_in'] / len(DATASET):>11.1f}  {gain:>7}")
    previous = d["hit_rate"]

FICTIONAL: example pool 8 solved tickets , measurement set 112 tickets
example format: Ornek : fatura ayrintisini nereden gorebilirim => fatura .

examples    label  hit rate  input tokens  per request     gain
       0        -    0.8571          6646         59.3        -
       1   fatura    0.8571          8438         75.3  +0.0000
       2    ariza    0.9018          9782         87.3  +0.0446
       3    sayac    0.9643         11574        103.3  +0.0625
       4   tarife    0.8929         13366        119.3  -0.0714
       5  kesinti    0.9196         15270        136.3  +0.0268
       6   fatura    0.9286         17398        155.3  +0.0089
       7    ariza    0.9107         18854        168.3  -0.0179
       8    sayac    0.9732         20310        181.3  +0.0625

The first row is zero-shot prompting: the prompt contains only the instruction and the format recipe, and the hit rate is 0.8571. The second row is one-shot prompting and its gain is zero. A single exemplar does nothing but raise the score of its own label; it says nothing about the other four labels and classifies the already correct tickets a little more correctly. Its cost is real: the input tokens per request rise from 59.3 to 75.3.

The gain starts with few-shot prompting. The second exemplar adds +0.0446, the third exemplar adds +0.0625, and the hit rate becomes 0.9643. Up to here adding exemplars pays for itself: from zero to three, +0.1071 hit rate for +4928 input tokens. After the fourth exemplar the chart changes direction. The fourth exemplar produces the largest single-step loss with −0.0714, the fifth and the sixth bring small gains, the seventh drops again, and the eighth brings it to the highest number with +0.0625. The mean of the five measurements between four and eight is 0.9250, that is, below the 0.9643 of three exemplars.

The saturation point in this chart is the third exemplar. What going from three to eight adds to the hit rate is +0.0089, and what it adds to the input tokens is +8736. The first three exemplars brought roughly twenty times more hit rate per token. The reason lies in what the exemplars teach: the first three examples introduce the tokens of three separate labels and correct the tickets that were wrongly matched before. The later examples either repeat a label that is already covered or teach the off-topic words in their own sentences; words such as “where” and “need” are unrelated to the label and push the score in the wrong direction. Past a certain point adding examples adds not information but noise, and the price of noise is also paid in tokens.

Same Examples, Different Order

ORDERS = [(0, 1, 2, 3), (3, 2, 1, 0), (1, 3, 0, 2), (2, 0, 3, 1), (0, 2, 1, 3), (3, 0, 2, 1)]
D = POOL[:4]
print("same four examples , only the order changes")
v = []
for s in ORDERS:
    d = measure([D[i] for i in s])
    v.append(d["hit_rate"])
    print(f"  {' > '.join(D[i]['topic'] for i in s):<34} hit rate {d['hit_rate']:.4f}"
          f"  input {d['tokens_in']}")
print(f"  order range {max(v) - min(v):.4f} , input tokens equal in every order")

print("\nvariability of the measure itself")
for n in (0, 3, 8):
    a, b = measure(POOL[:n]), measure(POOL[:n], seed=SEED + 1)
    print(f"  n={n} most likely choice : {a['hit_rate']:.4f} / {b['hit_rate']:.4f}"
          f"  difference {b['hit_rate'] - a['hit_rate']:+.4f} , failed calls {a['failed']}")
for n in (0, 3):
    a = measure(POOL[:n], temperature=0.6)
    b = measure(POOL[:n], seed=SEED + 1, temperature=0.6)
    print(f"  n={n} sampling on : {a['hit_rate']:.4f} / {b['hit_rate']:.4f}"
          f"  difference {b['hit_rate'] - a['hit_rate']:+.4f}")

s0, s3, s8 = measure(POOL[:0]), measure(POOL[:3]), measure(POOL[:8])
mid = sum(measure(POOL[:n])["hit_rate"] for n in range(4, 9)) / 5
print(f"\nzero -> three examples  : hit rate {s3['hit_rate'] - s0['hit_rate']:+.4f} ,"
      f" input tokens {s3['tokens_in'] - s0['tokens_in']:+d}")
print(f"three -> eight examples : hit rate {s8['hit_rate'] - s3['hit_rate']:+.4f} ,"
      f" input tokens {s8['tokens_in'] - s3['tokens_in']:+d}")
print(f"mean hit rate between four and eight {mid:.4f} , at three examples {s3['hit_rate']:.4f}")

same four examples , only the order changes
  fatura > ariza > sayac > tarife    hit rate 0.8929  input 13366
  tarife > sayac > ariza > fatura    hit rate 0.9196  input 13366
  ariza > tarife > fatura > sayac    hit rate 0.8929  input 13366
  sayac > fatura > tarife > ariza    hit rate 0.9196  input 13366
  fatura > sayac > ariza > tarife    hit rate 0.9196  input 13366
  tarife > fatura > sayac > ariza    hit rate 0.7679  input 13366
  order range 0.1518 , input tokens equal in every order

variability of the measure itself
  n=0 most likely choice : 0.8571 / 0.8571  difference +0.0000 , failed calls 3
  n=3 most likely choice : 0.9643 / 0.9554  difference -0.0089 , failed calls 3
  n=8 most likely choice : 0.9732 / 0.9464  difference -0.0268 , failed calls 3
  n=0 sampling on : 0.7321 / 0.8393  difference +0.1071
  n=3 sampling on : 0.8571 / 0.8750  difference +0.0179

zero -> three examples  : hit rate +0.1071 , input tokens +4928
three -> eight examples : hit rate +0.0089 , input tokens +8736
mean hit rate between four and eight 0.9250 , at three examples 0.9643

All six lines above contain the same four examples. The label set is the same, the instruction is the same, the format recipe is the same, the measurement set is the same, and the input tokens are 13366 in all six orders. The only thing that changes is the order in which the examples are written, and the hit rate wanders between 0.7679 and 0.9196. The range is 0.1518, that is, larger than the 0.1071 that going from zero to three examples adds. The same example set, with a cost identical to the last token, can produce the difference between a good prompt and a bad prompt.

The reason is that the simulator gives more weight to the last example. In the worst ordering the last example carries the fault label, and the but token in its sentence appears in all eight billing tickets of the measurement set; once weighted, it pulls those tickets to the fault side too. When the same example stands at the beginning of the list it does not have this effect. Order is a design decision and a free lever on the prompt: changing it brings no token cost.

The sub-block gives the variability of the measure itself. With the most likely choice, the difference between two runs is at most 0.0268; the number of failed calls is three in all three runs and comes from the rate limit. At this scale the +0.1071 gain of going from zero to three examples is four times the two-run difference and counts as measured; the +0.0089 gain of going from three to eight stays below the two-run difference and counts as unmeasured. When sampling is turned on the table turns around: at zero examples the difference between two runs is 0.1071, that is, exactly as large as the gain we want to measure. Someone who tries to measure the same number in a sampled run in a single shot sees the gain and the noise at the same size and cannot tell them apart. A prompt changed without measuring is not an improvement but a guess, and here measuring has a precondition: the variability of the measure itself must be smaller than the difference being sought.

Summary

• Zero-shot prompting gives 0.8571, one-shot prompting again gives 0.8571: a single exemplar raises the input tokens per request from 59.3 to 75.3 without adding anything to the hit rate.
• The gain accumulates in few-shot prompting and saturates at the third exemplar: from zero to three +0.1071 hit rate and +4928 input tokens, from three to eight +0.0089 hit rate and +8736 input tokens.
• The mean of the five measurements between four and eight exemplars is 0.9250, below the 0.9643 of three exemplars; the exemplars added after the saturation point carry noise, not information.
• When the same four exemplars are written with only their order changed, the hit rate wanders between 0.7679 and 0.9196. The range is 0.1518 and the input tokens are 13366 in all six orders: order is a free design lever.
• With the most likely choice the two-run difference is at most 0.0268; with sampling on it is 0.1071 at zero exemplars. If the gain being sought is smaller than the variability of the measure itself, the measurement counts as not made.

Next Step

In these two lessons everything written into the prompt was positive: do this, write in this format, look at these examples. In practice, a large part of prompts contains the opposite as well: sentences that say something must not be done. Do not use a label, do not give a date, do not speak with certainty, do not make things up. Intuition says these sentences are harmless: at worst they do not work. The next lesson takes the number of this intuition and finds the opposite. The same rule is written in two separate forms, a negative constraint and a positive statement, and the number of violations each one produces is printed side by side. Naming in the prompt the very thing the prohibition forbids has a cost of its own, and this cost shows up in the number.