Prompt Iteration

In the previous four lessons four design decisions were measured, and each was tried alone, one at a time. A real prompt is not written this way. A prompt is written, measured, changed in one place, measured again; this loop goes on until the number stops improving. The name of the loop is prompt iteration, and this lesson puts the loop itself under measurement.

To put it under measurement is to carry here a separation established in the Introduction to Machine Learning course of the previous curriculum: if you decide by looking at a number, that number is now the number of the set you decide on and does not represent general performance. There the separation was built with the training, validation and test sets. Here no model is trained, a prompt is searched; but the separation is the same separation. The set looked at in every iteration is the validation set, and the set never looked at is the test set.

• PD61. The model is a simulator, its seed is 20260218, and no real endpoint is called. The support tickets are fictional.
• PD62. The 120 fictional tickets are split in two: 30 records validation, 90 records test. The search sees only the validation set; the test set enters no decision.
• PD63. There are ten candidate changes. Five are relevant to the task (label set, format constraint, two examples, two more examples, positive statement), five are irrelevant (polite opening, write short, thanks, note line, reader note). The irrelevant ones change only the text of the prompt.
• PD64. The search is greedy: at every iteration all the remaining candidates are tried on validation, the one that gives the best number is chosen and added to the prompt permanently.
• PD65. The cost is written in number of calls. Every candidate tried spends as many calls as the size of the validation set, and these calls cannot be taken back.

The Two Numbers of Eight Iterations

# SIMULATOR -- FICTIONAL. This code does not call a real model endpoint; it is a
# language model simulator written with the standard library. The tickets are fictional too.
import math

SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW, MAX_OUT = 240, 24


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):                    # FICTIONAL tokenizer: pieces of at most five characters
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def prompt_trace(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
TEMPLATES = {"fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi",
                         "faturami odedim ama sistemde borc gorunuyor",
                         "fatura ayrintisini nereden gorebilirim"],
             "ariza": ["evde su akmiyor bir ariza var galiba",
                       "baglanti bozuk oldu ve hic calismiyor",
                       "ariza kaydi actim ama ekip gelmedi"],
             "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor",
                       "sayac degisimi icin randevu istiyorum",
                       "sayac uzerindeki endeks ile fatura ayni degil"],
             "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum",
                        "tarife degisikligi icin ne yapmam gerekiyor",
                        "birim ucret abone tipine gore degisiyor mu"],
             "kesinti": ["mahallede kesinti var ne zaman gelecek",
                        "planli kesinti duyurusu nereden takip edilir",
                        "aksamdan beri kesinti yasiyoruz"]}
EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim",
          "acil bilgi almam gerekiyor", "kayit numaram elimde yok"]

TICKETS = []
for i in range(120):
    r = rng(SEED + i)
    k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    m = TEMPLATES[k][int(r() * 3)]
    if r() < 0.34:
        m += " " + EXTRAS[int(r() * 4)]
    TICKETS.append({"id": 4001 + i, "text": m, "topic": k})

CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"]
CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura",
        "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza",
        "sayac": "sayac", "endek": "sayac", "okuma": "sayac",
        "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife",
        "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti",
        "borc": "odeme", "odedi": "odeme", "ucret": "odeme",
        "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet",
        "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"}
TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura",
         "ekip": "kesinti", "takip": "tarife", "giril": "fatura"}
FREE_FORM = ["talebiniz {k} konusuyla ilgili gorunuyor",
             "bu kayit {k} basligina girer , ilgili ekibe yonlendirildi",
             "kaydiniz alindi , konuyu inceleyip donus yapacagiz"]
CALL_COUNT = [0]


def parse(prompt):
    head, _, ticket = prompt.partition("Talep : ")
    o = []
    for part in head.split("Ornek : ")[1:]:
        m, _, e = part.partition(" => ")
        o.append((m.strip(), e.split(" .")[0].strip()))
    return o, ticket


def call(prompt, seed=SEED, temperature=0.6, max_out=MAX_OUT):
    CALL_COUNT[0] += 1
    tok = tokenize(prompt)
    empty = {"tokens_in": len(tok), "tokens_out": 0, "response": ""}
    if len(tok) > WINDOW:
        return dict(empty, status="context_overflow")
    if CALL_COUNT[0] % 37 == 0:
        return dict(empty, status="rate_limit")
    r = rng(seed ^ prompt_trace(prompt))
    examples, ticket = parse(prompt)
    tt = tokenize(ticket)
    cand = [k for k in CONCEPTS if k[:5] in tok]
    if len(cand) < 3:
        cand = CONCEPTS
    p = {k: 0.15 for k in cand}
    for t in tt:
        for table, w in ((CUES, 1.0), (TRAPS, 1.3)):
            if table.get(t) in p:
                p[table[t]] += w
    for j, (m, e) in enumerate(examples):
        if e in p:
            w = 0.5 + (j + 1) / len(examples)
            for t in set(tokenize(m)):
                if t in tt:
                    p[e] += 0.30 * w
    e = [math.exp(p[a] / temperature) for a in sorted(p)]
    k = sorted(p)[pick(r(), [v / sum(e) for v in e])]
    y = (f"etiket : {k}" if "bicim" in tok
         else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k))
    ot = tokenize(y)
    if len(ot) > max_out:
        ot, y = ot[:max_out], " ".join(ot[:max_out])
    return {"status": "ok", "tokens_in": len(tok), "tokens_out": len(ot), "response": y}


def read_label(y):
    for s in y.lower().split():
        if s in TOPICS:
            return s
    return None


VALIDATION, TEST = TICKETS[:30], TICKETS[30:]      # 30 validation , 90 test
BASE = "Su destek talebini yanitla . "
CHANGES = [
    ("label set", "Talebi su etiketlerden birine ata : fatura , ariza , sayac , "
                  "tarife , kesinti . "),
    ("format constraint", "Yanitin bicimi : etiket : <deger> . "),
    ("two examples", "Ornek : fatura ayrintisini nereden gorebilirim => fatura . "
                     "Ornek : ariza kaydi actim ama ekip gelmedi => ariza . "),
    ("two more examples", "Ornek : sayac okumasi yanlis girilmis endeks tutmuyor => sayac . "
                          "Ornek : hangi tarife basamagindayim ogrenmek istiyorum => tarife . "),
    ("positive statement", "Yanitinda yalnizca konu etiketi olsun . "),
    ("polite opening", "Lutfen dikkatli davran . "),
    ("write short", "Yanitini kisa tut . "),
    ("thanks", "Tesekkurler . "),
    ("note line", "Not : kayit sistemine islenecek . "),
    ("reader note", "Yanit destek ekibi tarafindan okunacak . "),
]


def measure(parts, dataset, seed=SEED):
    CALL_COUNT[0] = 0
    d = 0
    for x in dataset:
        s = call(BASE + "".join(parts) + "Talep : " + x["text"], seed=seed)
        d += read_label(s["response"]) == x["topic"]
    return d / len(dataset), CALL_COUNT[0]


def search(seed, steps=8):
    """At every iteration tries all candidates on VALIDATION and chooses the best."""
    chosen, remaining, calls, trace = [], list(CHANGES), 0, []
    for _ in range(steps):
        best, name = None, None
        for a, part in remaining:
            v, c = measure(chosen + [part], VALIDATION, seed)
            calls += c
            if best is None or v > best:
                best, name = v, (a, part)
        chosen.append(name[1])
        remaining = [x for x in remaining if x[0] != name[0]]
        dv, c1 = measure(chosen, VALIDATION, seed)
        sv, c2 = measure(chosen, TEST, seed)
        calls += c1 + c2
        trace.append((name[0], dv, sv, calls))
    return trace


d0, c0 = measure([], VALIDATION)
s0, c1 = measure([], TEST)
print("iteration  selected change       validation    test      gap  total calls")
print(f"{0:>9}  {'-':<20}  {d0:>10.4f}  {s0:.4f}  {d0 - s0:+.4f}  {c0 + c1:>11}")
TRACE = search(SEED)
for i, (name, dv, sv, cg) in enumerate(TRACE, 1):
    print(f"{i:>9}  {name:<20}  {dv:>10.4f}  {sv:.4f}  {dv - sv:+.4f}  {cg + c0 + c1:>11}")

iteration  selected change       validation    test      gap  total calls
        0  -                         0.4000  0.3778  +0.0222          120
        1  note line                 0.7000  0.5889  +0.1111          540
        2  format constraint         0.8000  0.5889  +0.2111          930
        3  two examples              0.8333  0.8556  -0.0222         1290
        4  label set                 0.9667  0.8556  +0.1111         1620
        5  positive statement        1.0000  0.9333  +0.0667         1920
        6  write short               0.9667  0.9111  +0.0556         2190
        7  two more examples         0.9667  0.8667  +0.1000         2430
        8  thanks                    0.9000  0.8111  +0.0889         2640

The validation column is the usual view of a search record: from 0.4000 to 1.0000, an uninterrupted rise up to the fifth iteration. Someone who looks at this column alone would say that the search has made the prompt fully accurate. The test column does not tell the same story. The highest test number is 0.9333 and it comes at the fifth iteration; the next three iterations bring it down to 0.8111. Every change made after the fifth iteration was chosen to protect validation and hurt on test.

The gap column carries the difference between the two sets. At the start it is +0.0222, that is, the two sets say almost the same thing. At the second iteration it rises to +0.2111: validation says 0.8000 while test is still 0.5889. At this point every decision made by looking at validation is speaking about a set that has nothing to do with it. The source of the gap is the smallness of the set – in thirty records the correction of one record moves the number by 0.0333, and the random correction of three records looks like an improvement.

The first row is the clearest example of this. The first change the search chooses is the note line: a sentence that adds no information to the task, that says nothing of the label set, the format or the example. It brought +0.3000 on validation and was the largest single step gain measured in the whole search. Three of the eight changes are of this kind. A greedy search chooses what raises the number; it does not ask why it raised the number.

The Reliability of the Search Itself

SECOND = search(SEED + 1)
print("same search , another run")
for i, ((a1, *_), (a2, d2, s2, _)) in enumerate(zip(TRACE, SECOND), 1):
    print(f"{i:>2}  {a1:<18} -> {a2:<18} validation {d2:.4f}  test {s2:.4f}")
print(f"changes the two searches chose in the same order: "
      f"{sum(a == b for (a, *_), (b, *_) in zip(TRACE, SECOND))}/8")

sel, fv, ft = [], [], []
for name, *_ in [(None,)] + TRACE:
    if name:
        sel.append(dict(CHANGES)[name])
    fv.append(abs(measure(sel, VALIDATION)[0] - measure(sel, VALIDATION, SEED + 1)[0]))
    ft.append(abs(measure(sel, TEST)[0] - measure(sel, TEST, SEED + 1)[0]))
gv = sum(fv) / len(fv)
print(f"\ntwo-run difference -- validation (30 records) mean {gv:.4f} , max {max(fv):.4f}")
print(f"two-run difference -- test (90 records) mean {sum(ft) / len(ft):.4f} ,"
      f" max {max(ft):.4f}")

vals, tests = [d0] + [x[1] for x in TRACE], [s0] + [x[2] for x in TRACE]
steps = [vals[i] - vals[i - 1] for i in range(1, 9)]
print(f"iterations whose step gain exceeds the two-run difference: "
      f"{sum(x > gv for x in steps)}/8 , mean step gain {sum(steps) / 8:+.4f}")
print(f"\nvalidation max {max(vals):.4f} (iteration {vals.index(max(vals))}) ,"
      f" test max {max(tests):.4f} (iteration {tests.index(max(tests))})")
print(f"5 -> 8 : validation {vals[5]:.4f} -> {vals[8]:.4f} , test {tests[5]:.4f} -> {tests[8]:.4f}")
print(f"gap max {max(d - s for d, s in zip(vals, tests)):+.4f} , at iteration 8 "
      f"{vals[8] - tests[8]:+.4f}")
irrelevant = ("polite opening", "write short", "thanks", "note line", "reader note")
print(f"search cost {TRACE[-1][3] + c0 + c1} calls , of the eight selected changes "
      f"{sum(a in irrelevant for a, *_ in TRACE)} irrelevant to the task")

same search , another run
 1  note line          -> write short        validation 0.7000  test 0.5667
 2  format constraint  -> label set          validation 0.6333  test 0.4444
 3  two examples       -> format constraint  validation 0.7667  test 0.7333
 4  label set          -> two examples       validation 0.9667  test 0.8778
 5  positive statement -> polite opening     validation 0.9667  test 0.9111
 6  write short        -> note line          validation 0.8667  test 0.7111
 7  two more examples  -> two more examples  validation 0.9000  test 0.8889
 8  thanks             -> reader note        validation 0.9333  test 0.8333
changes the two searches chose in the same order: 1/8

two-run difference -- validation (30 records) mean 0.1037 , max 0.1667
two-run difference -- test (90 records) mean 0.0728 , max 0.1778
iterations whose step gain exceeds the two-run difference: 2/8 , mean step gain +0.0625

validation max 1.0000 (iteration 5) , test max 0.9333 (iteration 5)
5 -> 8 : validation 1.0000 -> 0.9000 , test 0.9333 -> 0.8111
gap max +0.2111 , at iteration 8 +0.0889
search cost 2640 calls , of the eight selected changes 3 irrelevant to the task

The chart above reruns the search in the same scaffolding, changing only the seed. Of the eight steps, only one chooses the same change at the same place. The second run chooses “write short” at the first step, the label set at the second step and reaches “polite opening” at the fifth step. The two searches produced two different prompts; both look good on their own validation set, and both are the product of the same scaffolding. The search finds not what it looks for but the run.

The sub-section gives the number of this. On the validation set the difference between two runs of the same prompt is 0.1037 on average, 0.1667 at most. The mean gain of the search per step is +0.0625. In only two of the eight iterations the step gain is larger than its own measurement noise; the remaining six steps count as unmeasured by the rule of the course, and in spite of this they were added to the prompt permanently. On the test set the same difference is 0.0728 on average, that is, on a set three times larger the measure is somewhat calmer. A small validation set is both cheaper and produces by itself the noise that the search will chase.

The saturation point is the fifth iteration: validation hits 1.0000 there and cannot rise any further, and test takes its highest value there too. The next three iterations spend 720 extra calls and bring validation down to 0.9000 and test from 0.9333 to 0.8111. The total cost of the search is 2640 calls, and the larger part of these calls went to candidates that were not chosen. A prompt search is subject to the same law as a tuning search: as the budget grows, the gain on validation does not carry over to test. A prompt changed without measuring is not an improvement but a guess; if the measured set is wrong, measuring too is guessing.

Summary

• The greedy search carried the validation hit rate from 0.4000 to 1.0000 at the fifth iteration and pulled it back to 0.9000 at the eighth; the test hit rate in the same period rose from 0.3778 to its highest 0.9333 and fell to 0.8111.
• The gain saturated at the fifth iteration. The next three iterations spent 720 extra calls and lowered test by 0.1222; the gap between the two sets took its largest value at the second iteration, +0.2111.
• The first and largest step of the search (+0.3000) came from a sentence irrelevant to the task; three of the eight chosen changes add no information to the task.
• On validation the two-run difference is 0.1037 on average and the step gain +0.0625 on average: only two of the eight iterations are larger than their own measurement noise. On test the same difference is 0.0728.
• When the same search is run with another seed, only one of the eight steps chooses the same change at the same place; the total cost is 2640 calls.

Next Step

In eight lessons the prompt turned into a design object. It was split into its components, it gained a permanent instruction, it was opened up at the level of role and ticket, its format and length were determined, it took examples, its rules were written positively and it was iterated by measuring. At every step the only thing that changed was the text, and the cost of every step was written in tokens, calls or violations. Their common side is also clear: all of them arranged what to say to the model.

None of them asked the model how to think. In these eight lessons the model answered every ticket in a single move: it took the prompt, chose a label, wrote a sentence. There was no step in between. Yet a ticket such as calculating the difference between two billing periods and finding which tariff the overpayment was made under is not something to be solved in a single move; it asks for a few intermediate steps, and those steps must be produced by the model itself. The next topic takes this up and in its first lesson sets up the second claim of the course: asking for an intermediate step multiplies the tokens and does not pay off in every task. The same technique pays off in a multi-step calculation, while in a single-step classification it raises the cost and leaves the number where it was, and both are printed.