--- title: 'Prompt Iteration' source: 'https://academia.sh/en/courses/prompt-engineering/prompt-iteration' course: 'Prompt Engineering' language: en updated: '2026-10-21T05:02:33+00:00' license: 'CC BY-SA 4.0' objectives: - 'Builds the loop of improving by measuring with the separation of a validation set and a test set, and writes two numbers side by side at every iteration.' - 'Determines at which iteration the gain saturates and measures how the numbers of the two sets diverge after the saturation point.' --- # Prompt Iteration The loop of improving by measuring is itself put under measurement. Ten candidate changes are searched greedily over eight iterations on a validation set of 30 records: the validation hit rate rises from 0.4000 to 1.0000 and hits the ceiling at the fifth iteration, and the test hit rate reaches its highest value of 0.9333 at the same point and falls to 0.8111 at the eighth iteration. The mean of the two-run difference on validation is 0.1037 and the mean step gain is 0.0625: only two of the eight iterations are larger than their own measurement noise. When the search is repeated in another run, only one of the eight steps chooses the same change at the same place. The cost is 2640 calls, and three of the eight chosen changes are irrelevant to the task. In the previous four lessons four design decisions were measured, and each was tried alone, one at a time. A real prompt is not written this way. A prompt is written, measured, changed in one place, measured again; this loop goes on until the number stops improving. The name of the loop is **prompt iteration**, and this lesson puts the loop itself under measurement. To put it under measurement is to carry here a separation established in the Introduction to Machine Learning course of the previous curriculum: if you decide by **looking at** a number, that number is now the number of the set you decide on and does not represent general performance. There the separation was built with the training, validation and test sets. Here no model is trained, a **prompt is searched**; but the separation is the same separation. The set looked at in every iteration is the **validation** set, and the set never looked at is the **test** set. - **PD61.** The model is a **simulator**, its seed is **20260218**, and no real endpoint is called. The support tickets are **fictional**. - **PD62.** The 120 fictional tickets are split in two: **30 records validation**, **90 records test**. The search sees only the validation set; the test set enters no decision. - **PD63.** There are ten **candidate changes**. Five are relevant to the task (label set, format constraint, two examples, two more examples, positive statement), five are irrelevant (polite opening, write short, thanks, note line, reader note). The irrelevant ones change **only the text** of the prompt. - **PD64.** The search is **greedy**: at every iteration all the remaining candidates are tried on validation, the one that gives the best number is chosen and added to the prompt permanently. - **PD65.** The cost is written in **number of calls**. Every candidate tried spends as many calls as the size of the validation set, and these calls cannot be taken back. ## The Two Numbers of Eight Iterations ```python # SIMULATOR -- FICTIONAL. This code does not call a real model endpoint; it is a # language model simulator written with the standard library. The tickets are fictional too. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUT = 240, 24 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer: pieces of at most five characters b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def prompt_trace(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] TEMPLATES = {"fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"]} EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim", "acil bilgi almam gerekiyor", "kayit numaram elimde yok"] TICKETS = [] for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = TEMPLATES[k][int(r() * 3)] if r() < 0.34: m += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": m, "topic": k}) CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "borc": "odeme", "odedi": "odeme", "ucret": "odeme", "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet", "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"} TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} FREE_FORM = ["talebiniz {k} konusuyla ilgili gorunuyor", "bu kayit {k} basligina girer , ilgili ekibe yonlendirildi", "kaydiniz alindi , konuyu inceleyip donus yapacagiz"] CALL_COUNT = [0] def parse(prompt): head, _, ticket = prompt.partition("Talep : ") o = [] for part in head.split("Ornek : ")[1:]: m, _, e = part.partition(" => ") o.append((m.strip(), e.split(" .")[0].strip())) return o, ticket def call(prompt, seed=SEED, temperature=0.6, max_out=MAX_OUT): CALL_COUNT[0] += 1 tok = tokenize(prompt) empty = {"tokens_in": len(tok), "tokens_out": 0, "response": ""} if len(tok) > WINDOW: return dict(empty, status="context_overflow") if CALL_COUNT[0] % 37 == 0: return dict(empty, status="rate_limit") r = rng(seed ^ prompt_trace(prompt)) examples, ticket = parse(prompt) tt = tokenize(ticket) cand = [k for k in CONCEPTS if k[:5] in tok] if len(cand) < 3: cand = CONCEPTS p = {k: 0.15 for k in cand} for t in tt: for table, w in ((CUES, 1.0), (TRAPS, 1.3)): if table.get(t) in p: p[table[t]] += w for j, (m, e) in enumerate(examples): if e in p: w = 0.5 + (j + 1) / len(examples) for t in set(tokenize(m)): if t in tt: p[e] += 0.30 * w e = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [v / sum(e) for v in e])] y = (f"etiket : {k}" if "bicim" in tok else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k)) ot = tokenize(y) if len(ot) > max_out: ot, y = ot[:max_out], " ".join(ot[:max_out]) return {"status": "ok", "tokens_in": len(tok), "tokens_out": len(ot), "response": y} def read_label(y): for s in y.lower().split(): if s in TOPICS: return s return None VALIDATION, TEST = TICKETS[:30], TICKETS[30:] # 30 validation , 90 test BASE = "Su destek talebini yanitla . " CHANGES = [ ("label set", "Talebi su etiketlerden birine ata : fatura , ariza , sayac , " "tarife , kesinti . "), ("format constraint", "Yanitin bicimi : etiket : . "), ("two examples", "Ornek : fatura ayrintisini nereden gorebilirim => fatura . " "Ornek : ariza kaydi actim ama ekip gelmedi => ariza . "), ("two more examples", "Ornek : sayac okumasi yanlis girilmis endeks tutmuyor => sayac . " "Ornek : hangi tarife basamagindayim ogrenmek istiyorum => tarife . "), ("positive statement", "Yanitinda yalnizca konu etiketi olsun . "), ("polite opening", "Lutfen dikkatli davran . "), ("write short", "Yanitini kisa tut . "), ("thanks", "Tesekkurler . "), ("note line", "Not : kayit sistemine islenecek . "), ("reader note", "Yanit destek ekibi tarafindan okunacak . "), ] def measure(parts, dataset, seed=SEED): CALL_COUNT[0] = 0 d = 0 for x in dataset: s = call(BASE + "".join(parts) + "Talep : " + x["text"], seed=seed) d += read_label(s["response"]) == x["topic"] return d / len(dataset), CALL_COUNT[0] def search(seed, steps=8): """At every iteration tries all candidates on VALIDATION and chooses the best.""" chosen, remaining, calls, trace = [], list(CHANGES), 0, [] for _ in range(steps): best, name = None, None for a, part in remaining: v, c = measure(chosen + [part], VALIDATION, seed) calls += c if best is None or v > best: best, name = v, (a, part) chosen.append(name[1]) remaining = [x for x in remaining if x[0] != name[0]] dv, c1 = measure(chosen, VALIDATION, seed) sv, c2 = measure(chosen, TEST, seed) calls += c1 + c2 trace.append((name[0], dv, sv, calls)) return trace d0, c0 = measure([], VALIDATION) s0, c1 = measure([], TEST) print("iteration selected change validation test gap total calls") print(f"{0:>9} {'-':<20} {d0:>10.4f} {s0:.4f} {d0 - s0:+.4f} {c0 + c1:>11}") TRACE = search(SEED) for i, (name, dv, sv, cg) in enumerate(TRACE, 1): print(f"{i:>9} {name:<20} {dv:>10.4f} {sv:.4f} {dv - sv:+.4f} {cg + c0 + c1:>11}") ``` ``` iteration selected change validation test gap total calls 0 - 0.4000 0.3778 +0.0222 120 1 note line 0.7000 0.5889 +0.1111 540 2 format constraint 0.8000 0.5889 +0.2111 930 3 two examples 0.8333 0.8556 -0.0222 1290 4 label set 0.9667 0.8556 +0.1111 1620 5 positive statement 1.0000 0.9333 +0.0667 1920 6 write short 0.9667 0.9111 +0.0556 2190 7 two more examples 0.9667 0.8667 +0.1000 2430 8 thanks 0.9000 0.8111 +0.0889 2640 ``` The validation column is the usual view of a search record: **from 0.4000 to 1.0000**, an uninterrupted rise up to the fifth iteration. Someone who looks at this column alone would say that the search has made the prompt fully accurate. The test column does not tell the same story. The highest test number is **0.9333** and it comes at the fifth iteration; the next three iterations bring it down to **0.8111**. Every change made after the fifth iteration was chosen to protect validation and hurt on test. The gap column carries the difference between the two sets. At the start it is **+0.0222**, that is, the two sets say almost the same thing. At the second iteration it rises to **+0.2111**: validation says 0.8000 while test is still 0.5889. At this point every decision made by looking at validation is speaking about a set that has nothing to do with it. The source of the gap is the smallness of the set -- in thirty records the correction of one record moves the number by 0.0333, and the random correction of three records looks like an improvement. The first row is the clearest example of this. The first change the search chooses is the **note line**: a sentence that adds no information to the task, that says nothing of the label set, the format or the example. It brought **+0.3000** on validation and was the largest single step gain measured in the whole search. Three of the eight changes are of this kind. A greedy search chooses what raises the number; it does not ask why it raised the number. ## The Reliability of the Search Itself ```python SECOND = search(SEED + 1) print("same search , another run") for i, ((a1, *_), (a2, d2, s2, _)) in enumerate(zip(TRACE, SECOND), 1): print(f"{i:>2} {a1:<18} -> {a2:<18} validation {d2:.4f} test {s2:.4f}") print(f"changes the two searches chose in the same order: " f"{sum(a == b for (a, *_), (b, *_) in zip(TRACE, SECOND))}/8") sel, fv, ft = [], [], [] for name, *_ in [(None,)] + TRACE: if name: sel.append(dict(CHANGES)[name]) fv.append(abs(measure(sel, VALIDATION)[0] - measure(sel, VALIDATION, SEED + 1)[0])) ft.append(abs(measure(sel, TEST)[0] - measure(sel, TEST, SEED + 1)[0])) gv = sum(fv) / len(fv) print(f"\ntwo-run difference -- validation (30 records) mean {gv:.4f} , max {max(fv):.4f}") print(f"two-run difference -- test (90 records) mean {sum(ft) / len(ft):.4f} ," f" max {max(ft):.4f}") vals, tests = [d0] + [x[1] for x in TRACE], [s0] + [x[2] for x in TRACE] steps = [vals[i] - vals[i - 1] for i in range(1, 9)] print(f"iterations whose step gain exceeds the two-run difference: " f"{sum(x > gv for x in steps)}/8 , mean step gain {sum(steps) / 8:+.4f}") print(f"\nvalidation max {max(vals):.4f} (iteration {vals.index(max(vals))}) ," f" test max {max(tests):.4f} (iteration {tests.index(max(tests))})") print(f"5 -> 8 : validation {vals[5]:.4f} -> {vals[8]:.4f} , test {tests[5]:.4f} -> {tests[8]:.4f}") print(f"gap max {max(d - s for d, s in zip(vals, tests)):+.4f} , at iteration 8 " f"{vals[8] - tests[8]:+.4f}") irrelevant = ("polite opening", "write short", "thanks", "note line", "reader note") print(f"search cost {TRACE[-1][3] + c0 + c1} calls , of the eight selected changes " f"{sum(a in irrelevant for a, *_ in TRACE)} irrelevant to the task") ``` ``` same search , another run 1 note line -> write short validation 0.7000 test 0.5667 2 format constraint -> label set validation 0.6333 test 0.4444 3 two examples -> format constraint validation 0.7667 test 0.7333 4 label set -> two examples validation 0.9667 test 0.8778 5 positive statement -> polite opening validation 0.9667 test 0.9111 6 write short -> note line validation 0.8667 test 0.7111 7 two more examples -> two more examples validation 0.9000 test 0.8889 8 thanks -> reader note validation 0.9333 test 0.8333 changes the two searches chose in the same order: 1/8 two-run difference -- validation (30 records) mean 0.1037 , max 0.1667 two-run difference -- test (90 records) mean 0.0728 , max 0.1778 iterations whose step gain exceeds the two-run difference: 2/8 , mean step gain +0.0625 validation max 1.0000 (iteration 5) , test max 0.9333 (iteration 5) 5 -> 8 : validation 1.0000 -> 0.9000 , test 0.9333 -> 0.8111 gap max +0.2111 , at iteration 8 +0.0889 search cost 2640 calls , of the eight selected changes 3 irrelevant to the task ``` The chart above reruns the search **in the same scaffolding, changing only the seed**. Of the eight steps, **only one** chooses the same change at the same place. The second run chooses "write short" at the first step, the label set at the second step and reaches "polite opening" at the fifth step. The two searches produced two different prompts; both look good on their own validation set, and both are the product of the same scaffolding. **The search finds not what it looks for but the run.** The sub-section gives the number of this. On the validation set the difference between two runs of the same prompt is **0.1037** on average, 0.1667 at most. The mean gain of the search per step is **+0.0625**. In **only two** of the eight iterations the step gain is larger than its own measurement noise; the remaining six steps count as **unmeasured** by the rule of the course, and in spite of this they were added to the prompt permanently. On the test set the same difference is **0.0728** on average, that is, on a set three times larger the measure is somewhat calmer. A small validation set is both cheaper and produces by itself the noise that the search will chase. The saturation point is the **fifth iteration**: validation hits 1.0000 there and cannot rise any further, and test takes its highest value there too. The next three iterations spend **720 extra calls** and bring validation down to 0.9000 and test from 0.9333 to 0.8111. The total cost of the search is **2640 calls**, and the larger part of these calls went to candidates that were not chosen. A prompt search is subject to the same law as a tuning search: as the budget grows, the gain on validation does not carry over to test. **A prompt changed without measuring is not an improvement but a guess**; if the measured set is wrong, measuring too is guessing. ## Summary - The greedy search carried the validation hit rate **from 0.4000** to **1.0000** at the fifth iteration and pulled it back to 0.9000 at the eighth; the test hit rate in the same period rose **from 0.3778** to its highest **0.9333** and fell to **0.8111**. - The gain **saturated at the fifth iteration**. The next three iterations spent 720 extra calls and lowered test by 0.1222; the gap between the two sets took its largest value at the second iteration, **+0.2111**. - The first and largest step of the search (**+0.3000**) came from a sentence irrelevant to the task; **three** of the eight chosen changes add no information to the task. - On validation the two-run difference is **0.1037** on average and the step gain **+0.0625** on average: only **two** of the eight iterations are larger than their own measurement noise. On test the same difference is **0.0728**. - When the same search is run with another seed, **only one** of the eight steps chooses the same change at the same place; the total cost is **2640 calls**. ## Next Step In eight lessons the prompt turned into a design object. It was split into its components, it gained a permanent instruction, it was opened up at the level of role and ticket, its format and length were determined, it took examples, its rules were written positively and it was iterated by measuring. At every step the only thing that changed was the **text**, and the cost of every step was written in tokens, calls or violations. Their common side is also clear: all of them arranged **what to say** to the model. None of them asked the model **how to think**. In these eight lessons the model answered every ticket in a single move: it took the prompt, chose a label, wrote a sentence. There was no step in between. Yet a ticket such as calculating the difference between two billing periods and finding which tariff the overpayment was made under is not something to be solved in a single move; it asks for a few intermediate steps, and those steps must be produced by the model itself. The next topic takes this up and in its first lesson sets up the second claim of the course: asking for an intermediate step **multiplies the tokens** and does not pay off in every task. The same technique pays off in a multi-step calculation, while in a single-step classification it raises the cost and leaves the number where it was, and both are printed.