--- title: 'Automatic Prompt Optimization' source: 'https://academia.sh/en/courses/prompt-engineering/automatic-prompt-optimization' course: 'Prompt Engineering' language: en updated: '2026-10-21T05:02:34+00:00' license: 'CC BY-SA 4.0' objectives: - 'Builds prompt search as a loop: produces the candidates mechanically, scores all of them with a single criterion and determines the one chosen for each budget.' - 'Measures how the validation and test scores come apart as the search budget grows and computes the optimism margin of the chosen candidate.' - 'Writes the cost of the search in number of calls and compares the difference obtained with the difference of the same candidate between two runs.' --- # Automatic Prompt Optimization Instead of being written by hand, the prompt is searched with a criterion: ten candidates are produced mechanically from four parts, and the fictional support tickets are split into a 40-record validation set and an 80-record test set. As the search budget goes from one candidate to ten, the best score on validation rises from 0.7750 to 0.9000 and the test score of the same candidate falls from 0.6750 to 0.6500: validation gain +0.1250, test gain -0.0250, and the chosen candidate's optimism margin 0.2500. The search goes from 40 calls to 400 calls, the candidate that is truly best on the test (0.8000) is never chosen, and in the second run the search chooses a different candidate. In the previous lesson three prompts were written by hand, all three were measured on the same set and the best was chosen by eye. The basis of that choice was a single run, and in the second run the ranking reversed. The hand-done part of this job is actually narrow: producing the candidates is a combination job, scoring is a loop, choosing is a comparison. All three can be written. When they are written, what comes out is **automatic prompt optimization** — searching for the prompt with a criterion. The search itself is not the subject of this lesson. The procedure of choosing the best over a set of candidates and **the optimism of the best score on the validation set as the tuning budget grows** were measured in the hyperparameter search lesson of the Supervised Learning course and are not repeated. What is searched here is not a tuning value but a **text**, and what gives the score is a nondeterministic component. The lesson's single question is this: as the search budget grows, does the number gained on validation carry over to a test set that has never been seen. - **RT69.** The code does not call a real model endpoint; it calls a **simulator** written with the standard library. The support tickets and the topics are **fictional**. - **RT70.** **The number of candidates does not exceed 10.** The candidates are produced mechanically from four parts: the role sentence, the prefix sentence, the order of the label list and the format sentence. - **RT71.** The set is split in two: **validation 40 records**, **test 80 records**. The test set is never seen during the search and no choice is made by looking at it. - **RT72.** The criterion is **label accuracy**; if a label cannot be read from the response, the record counts as wrong. - **RT73.** Search rule: when the budget is $b$, the one with the highest validation among the **first $b$ candidates** is chosen. In a tie, the one that comes first in the list wins. - **RT74.** The cost of the search is written **in number of calls**. Every candidate is run on the whole validation set; for budget $b$ the cost is $b \times 40$ calls. - **RT75.** The **optimism margin** is the difference between the chosen candidate's validation score and its test score. - **RT76.** Two runs are printed. With seed 20260219 the search is repeated from scratch; whether the chosen candidate changed and how much the test score of the same candidate wobbled is written. ## Ten Candidates, Two Sets ```python # SIMULATOR -- FICTIONAL. This code does not call a real model endpoint; it calls a language model # simulator written with the standard library. The tickets are fictional too. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUT_TOKENS = 240, 24 # context window and largest output, in tokens def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): # FICTIONAL tokenizer: pieces of at most five characters out = [] for piece in text.lower().replace(",", " , ").split(): while len(piece) > 5: out.append(piece[:5]) piece = piece[5:] out.append(piece) return out def trace_id(text): # a trace tied to the content of the prompt, independent of the run h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] PATTERNS = { "fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"], } EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim", "acil bilgi almam gerekiyor", "kayit numaram elimde yok"] TICKETS = [] for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] text = PATTERNS[k][int(r() * 3)] if r() < 0.34: text += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": text, "topic": k}) CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CUE = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "borc": "odeme", "odedi": "odeme", "ucret": "odeme", "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet", "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"} TRAP = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} # misleading links FREE_FORM = ["talebiniz {k} konusuyla ilgili gorunuyor", "bu kayit {k} basligina girer , ilgili ekibe yonlendirildi", "kaydiniz alindi , konuyu inceleyip donus yapacagiz"] COUNTER = [0] def call(prompt, seed=SEED, temperature=0.6, max_out=MAX_OUT_TOKENS): COUNTER[0] += 1 toks = tokenize(prompt) if len(toks) > WINDOW: return {"status": "context_overflow", "input": len(toks), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: return {"status": "rate_limit", "input": len(toks), "output": 0, "response": ""} r = rng(seed ^ trace_id(prompt)) if len(toks) > 90 and r() < 0.04: return {"status": "timeout", "input": len(toks), "output": 0, "response": ""} candidates = [k for k in CONCEPTS if k[:5] in toks] # concepts named in the prompt if len(candidates) < 3: candidates = CONCEPTS p = {k: 0.15 for k in candidates} for t in toks: for table, w in ((CUE, 1.0), (TRAP, 1.3)): if table.get(t) in p: p[table[t]] += w if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) # most likely choice else: e = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [v / sum(e) for v in e])] y = k if "tek" in toks else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k) out = tokenize(y) if len(out) > max_out: # stop condition out = out[:max_out] y = " ".join(out) return {"status": "done", "input": len(toks), "output": len(out), "response": y} def read_label(y): # extracting the label from the response for s in y.lower().split(): if s in TOPICS: return s return None ROLES = ["", "Sen destek masasi gorevlisisin . ", "Kayit gorevlisi olarak yanitla . "] PREFIXES = ["", "Once talebin hangi islem alanina girdigini belirle , sonra ", "Ornek: endeks tutmuyor -> sayac . "] LISTS = ["fatura , ariza , sayac , tarife , kesinti", "kesinti , tarife , sayac , ariza , fatura", "ariza , kesinti , fatura , sayac , tarife"] FORMATS = ["Yanit olarak tek etiket yaz .", "Tek etiket yaz , aciklama ekleme .", "Cevabin tek sozcuk olsun , listedeki etiketlerden biri ."] CANDIDATES, PARTS = [], [] r = rng(SEED + 777) while len(CANDIDATES) < 10: # candidates are produced mechanically, the seed is fixed d = [int(r() * 3) for _ in range(4)] y = (ROLES[d[0]] + PREFIXES[d[1]] + "Talebi su etiketlerden birine ata: " + LISTS[d[2]] + " . " + FORMATS[d[3]] + " Talep: ") if y not in CANDIDATES: CANDIDATES.append(y) PARTS.append("".join(str(x) for x in d)) VAL, TEST = TICKETS[:40], TICKETS[40:] def score_prompt(instruction, subset, seed=SEED): right = 0 for x in subset: right += read_label(call(instruction + x["text"], seed=seed)["response"]) == x["topic"] return right / len(subset) COUNTER[0] = 0 VAL_SCORES = [score_prompt(y, VAL) for y in CANDIDATES] SEARCH_CALLS = COUNTER[0] TEST_SCORES = [score_prompt(y, TEST) for y in CANDIDATES] print(f"{'candidate':>9} {'parts':>6} {'tokens':>8} {'validation':>10} {'test':>8} " f"{'diff':>8}") for i, y in enumerate(CANDIDATES): print(f"{i + 1:>9d} {PARTS[i]:>6} {len(tokenize(y)):>8d} {VAL_SCORES[i]:>10.4f} " f"{TEST_SCORES[i]:>8.4f} {TEST_SCORES[i] - VAL_SCORES[i]:>+8.4f}") print(f"\nvalidation set {len(VAL)} records, test set {len(TEST)} records") print(f"search calls {SEARCH_CALLS}, validation mean {sum(VAL_SCORES) / 10:.4f}, " f"test mean {sum(TEST_SCORES) / 10:.4f}") ``` ``` candidate parts tokens validation test diff 1 2110 53 0.7750 0.6750 -0.1000 2 2201 51 0.7250 0.7250 +0.0000 3 1210 50 0.7500 0.6500 -0.1000 4 1000 41 0.6500 0.8000 +0.1500 5 0212 47 0.4250 0.4625 +0.0375 6 1101 56 0.9000 0.6500 -0.2500 7 0121 47 0.6000 0.6750 +0.0750 8 0202 47 0.5250 0.6000 +0.0750 9 1020 41 0.7750 0.7375 -0.0375 10 0100 45 0.7750 0.6875 -0.0875 validation set 40 records, test set 80 records search calls 400, validation mean 0.6900, test mean 0.6663 ``` The last line closes a wrong explanation from the start. The means of the two sets are **0.6900** and **0.6663**; there is no systematic gulf between them. That is, the test set is not harder and its distribution is not different. Even so, the `diff` column goes both ways at once and reaches a size of **0.2500**. The split is not between the sets, it is **per candidate**. The sixth candidate is first on validation by a wide margin with **0.9000**, and third from last on the test with **0.6500**. The fourth candidate is the exact opposite: low on validation with 0.6500, first on the test with **0.8000**. In a set of forty records a single record is worth 0.0250; the **0.1250** difference that separates the sixth candidate from the first one is nothing more than where five records fell. The `parts` column gives the four digits of the candidate: the role sentence, the prefix sentence, the order of the label list, the format sentence. The fourth candidate is `1000`, the ninth candidate is `1020`; they differ only in the third digit. The same role sentence, the same absence of a prefix, the same format sentence, the same five labels — the only difference is the **order the labels are written in** and the test scores are **0.8000** and **0.7375**. The score wobbling by 0.0625 with the information the prompt carries unchanged already opens the question of what this search measures. ## As the Search Budget Grows ```python def select_best(vs): # budget b: the one with the highest validation among the first b return [max(range(b), key=lambda i: vs[i]) for b in range(1, len(vs) + 1)] CHOSEN = select_best(VAL_SCORES) print(f"{'budget':>6} {'chosen':>6} {'validation':>10} {'test':>8} {'search calls':>12}") for b, i in enumerate(CHOSEN, 1): print(f"{b:>6d} {i + 1:>6d} {VAL_SCORES[i]:>10.4f} {TEST_SCORES[i]:>8.4f} {b * len(VAL):>12d}") first, last = CHOSEN[0], CHOSEN[-1] print(f"\nbudget 1 -> 10: validation {VAL_SCORES[last] - VAL_SCORES[first]:+.4f}, " f"test {TEST_SCORES[last] - TEST_SCORES[first]:+.4f}") print(f"optimism margin of the chosen candidate {VAL_SCORES[last] - TEST_SCORES[last]:+.4f}") best_test = max(range(10), key=lambda i: TEST_SCORES[i]) print(f"best candidate on test {best_test + 1} ({TEST_SCORES[best_test]:.4f}); its validation " f"{VAL_SCORES[best_test]:.4f}, the search never chose it") COUNTER[0] = 0 VAL_SCORES2 = [score_prompt(y, VAL, SEED + 1) for y in CANDIDATES] TEST_SCORES2 = [score_prompt(y, TEST, SEED + 1) for y in CANDIDATES] last2 = select_best(VAL_SCORES2)[-1] print(f"\nsecond run (seed {SEED + 1}): chosen candidate {last2 + 1}, validation " f"{VAL_SCORES2[last2]:.4f}, test {TEST_SCORES2[last2]:.4f}") print(f"test of candidate {last + 1} over two runs {TEST_SCORES[last]:.4f} -> " f"{TEST_SCORES2[last]:.4f} ({TEST_SCORES2[last] - TEST_SCORES[last]:+.4f})") print(f"is the test ranking of the ten candidates the same in both runs: " f"{sorted(range(10), key=lambda i: -TEST_SCORES[i]) == sorted(range(10), key=lambda i: -TEST_SCORES2[i])}") ``` ``` budget chosen validation test search calls 1 1 0.7750 0.6750 40 2 1 0.7750 0.6750 80 3 1 0.7750 0.6750 120 4 1 0.7750 0.6750 160 5 1 0.7750 0.6750 200 6 6 0.9000 0.6500 240 7 6 0.9000 0.6500 280 8 6 0.9000 0.6500 320 9 6 0.9000 0.6500 360 10 6 0.9000 0.6500 400 budget 1 -> 10: validation +0.1250, test -0.0250 optimism margin of the chosen candidate +0.2500 best candidate on test 4 (0.8000); its validation 0.6500, the search never chose it second run (seed 20260219): chosen candidate 9, validation 0.7500, test 0.7875 test of candidate 6 over two runs 0.6500 -> 0.8500 (+0.2000) is the test ranking of the ten candidates the same in both runs: False ``` Two columns of the table move in opposite directions. As the budget goes from one to ten, the validation column rises **from 0.7750 to 0.9000** — the search works, does its job, finds a higher number with every new candidate. On the same rows the test column falls **from 0.6750 to 0.6500**. The validation gain is **+0.1250**, the test gain is **−0.0250**. Raising the search budget ten times gained nothing on data that has never been seen; it brought a small loss. The cost is in the same table and beyond dispute: **from 40 calls to 400 calls**. An extra 360 calls were spent and the only thing obtained in return was **a 0.1250 rise in the number to be reported**. The chosen candidate's **optimism margin is 0.2500**: the prompt that reads 0.9000 on validation gives 0.6500 on the records never seen. Delivering this prompt as "the 0.9000 prompt" means making up a quarter of the measure. The third row shows the blindness of the search. The candidate that is truly best on the test is the fourth (**0.8000**) and the search did not choose it under any budget; because it scored 0.6500 on validation, it was eliminated in the very first comparison. The search did not find the good prompt, it found the prompt that **fits the validation set best**. ## Two Runs and What the Search Measures The last three rows probe the ground of the whole exercise. When the seed is changed and the search is run from scratch, the chosen candidate slides **from the sixth to the ninth**: the same candidates, the same sets, the same rule, a different winner. The ninth candidate differed from the fourth, which was first on the test, only by the label order. Moreover, the sixth candidate's test score wobbles **from 0.6500 to 0.8500**, that is by **+0.2000**. The validation difference the search won over ten candidates was 0.1250; **the wobble of a single candidate from run to run is above that**. The last row completes this: the test ranking of the ten candidates is not the same in the two runs. The conclusion drawn from this does not forbid the search, it corrects its yardstick. The best score on the validation set is not the quality of a prompt, it is **the residue of a search budget**; it grows optimistic as the budget grows, and this optimism is not visible unless it is measured with a separate number on a set that has never been touched. The same holds for the second run of a nondeterministic component. The curriculum's rule takes its final form here: **a prompt changed without measurement is not an improvement but a guess** — and a prompt measured on a single set in a single run is the same. ## Summary - Ten candidates are produced mechanically from four parts; the validation set is 40, the test set 80 records. The means of the two sets are **0.6900** and **0.6663**, so the split is not between the sets but per candidate. - As the search budget goes from one to ten, validation rises **from 0.7750 to 0.9000** and the test of the same candidate falls **from 0.6750 to 0.6500**: **−0.0250** in return for **+0.1250**. - The cost is written in number of calls: the search goes **from 40 calls to 400 calls**. - The chosen candidate's **optimism margin is 0.2500**. The candidate that is truly best on the test (0.8000) is not chosen under any budget because it scored 0.6500 on validation. - In the second run the search chooses a different candidate and the sixth candidate's test wobbles by **+0.2000** — the run difference of a single candidate is larger than the difference the search won over ten candidates. ## Next Step Over sixteen lessons what to say to the prompt was arranged and the model was asked how to think: instruction, role, format, example, negative constraint, step-by-step solving, multiple samples, branching, stepping back, action field, ensemble and now search. Under all of them sat something silent — the response coming from the model was **free text**. In every lesson the label was extracted from inside the response with a reading procedure; a response that could not be read was counted wrong and that share was counted separately. Every improvement made on the prompt side amounted to **asking for** the format of the output. The next topic moves to this side: the settings the response is generated with, where the response stops, imposing the format, validating the incoming output against the schema and the business rule, and how much of the response that gets caught by validation is rescued by retrying.