--- title: 'Prompt Ensemble' source: 'https://academia.sh/en/courses/prompt-engineering/prompt-ensemble' course: 'Prompt Engineering' language: en updated: '2026-10-21T05:02:34+00:00' license: 'CC BY-SA 4.0' objectives: - 'Measures the label accuracy of three separate instructions on the same task in two runs and shows that the ranking changes from run to run.' - 'Counts the overlap of the errors of different prompts: computes the number of records on which all three are wrong at once and the ceiling this determines.' - 'Reads the gain of the majority vote against the ceiling and the two-run difference, and writes its cost as number of calls and tokens per correct label.' --- # Prompt Ensemble The same 120 fictional support tickets are examined with three separate instructions, and the assumption the ensemble rests on is tested: do different prompts make different errors. The three prompts alone give 0.7750, 0.6583 and 0.7833, on nine records all three are wrong at once and the ceiling comes out at 0.9250. The majority vote stays at 0.8000: 0.1250 below the ceiling, only 0.0167 above the best single prompt, and this gain is below the 0.0333 to 0.1333 difference the same prompts show between two runs. The cost is written in number of calls: 120 calls become 360 calls, and 61.1 tokens per correct label become 180.7 tokens. The previous lesson measured how far a single prompt can be stretched: a model class that produces invisible intermediate steps before the response spent more tokens on the same task, and it was written that the difference stayed unmeasurable in one task. What the six lessons have in common is this — there was always **a single prompt**. That prompt was lengthened, forced into step-by-step solving, branched, split in two. Having **several separate prompts** look at the same question was never tried. This is the scaffolding of this lesson: the same task, the same simulator, the same set; three separately written instructions and, at the end, a vote. Its name is the **prompt ensemble**. The procedure of the majority vote was measured in the ensemble methods lesson of the Supervised Learning course and is not repeated. Two things are added: writing the cost **in number of calls** and testing the assumption the ensemble rests on. That assumption is this — different prompts make **different errors**. The number of this lesson is how far the assumption holds: on how many records all three prompts are wrong **at once**. - **RT61.** The task, the set and the simulator are fixed. The only thing that changes is the **text of the instruction**; all three instructions carry the same five labels and the same format constraint. - **RT62.** The code does not call a real model endpoint; it calls a **simulator** written with the standard library. The support tickets and the topics are **fictional**. - **RT63.** The three instructions carry three separate design decisions: the plain label set, stating a role and asking for the operation area first. - **RT64.** The criterion is **label accuracy**. If a label cannot be read from the response, the record counts as wrong. - **RT65.** The ensemble's decision is the **majority vote**. If all three labels differ, the label of the first prompt holds; the number of these cases is printed separately. - **RT66.** **Overlap** is this: if all three prompts are wrong on a record at once, that record is one the ensemble cannot reach. The share of the remaining records is the ensemble's **ceiling**. - **RT67.** The cost is written **in number of calls**: a three-member ensemble spends three calls for each record. The unit cost per token and per correct label is put next to it. - **RT68.** Every measure is printed in **two runs** (seeds 20260218 and 20260219). If the ensemble's gain is smaller than the two-run difference it counts as **unmeasured**. ## Three Prompts Looking at the Same Task ```python # SIMULATOR -- FICTIONAL. This code does not call a real model endpoint; it calls a language model # simulator written with the standard library. The tickets are fictional too. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUT_TOKENS = 240, 24 # context window and largest output, in tokens def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): # FICTIONAL tokenizer: pieces of at most five characters out = [] for piece in text.lower().replace(",", " , ").split(): while len(piece) > 5: out.append(piece[:5]) piece = piece[5:] out.append(piece) return out def trace_id(text): # a trace tied to the content of the prompt, independent of the run h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] PATTERNS = { "fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi", "faturami odedim ama sistemde borc gorunuyor", "fatura ayrintisini nereden gorebilirim"], "ariza": ["evde su akmiyor bir ariza var galiba", "baglanti bozuk oldu ve hic calismiyor", "ariza kaydi actim ama ekip gelmedi"], "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor", "sayac degisimi icin randevu istiyorum", "sayac uzerindeki endeks ile fatura ayni degil"], "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum", "tarife degisikligi icin ne yapmam gerekiyor", "birim ucret abone tipine gore degisiyor mu"], "kesinti": ["mahallede kesinti var ne zaman gelecek", "planli kesinti duyurusu nereden takip edilir", "aksamdan beri kesinti yasiyoruz"], } EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim", "acil bilgi almam gerekiyor", "kayit numaram elimde yok"] TICKETS = [] for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] text = PATTERNS[k][int(r() * 3)] if r() < 0.34: text += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": text, "topic": k}) CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"] CUE = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti", "borc": "odeme", "odedi": "odeme", "ucret": "odeme", "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet", "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"} TRAP = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} # misleading links FREE_FORM = ["talebiniz {k} konusuyla ilgili gorunuyor", "bu kayit {k} basligina girer , ilgili ekibe yonlendirildi", "kaydiniz alindi , konuyu inceleyip donus yapacagiz"] COUNTER = [0] def call(prompt, seed=SEED, temperature=0.6, max_out=MAX_OUT_TOKENS): COUNTER[0] += 1 toks = tokenize(prompt) if len(toks) > WINDOW: return {"status": "context_overflow", "input": len(toks), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: return {"status": "rate_limit", "input": len(toks), "output": 0, "response": ""} r = rng(seed ^ trace_id(prompt)) if len(toks) > 90 and r() < 0.04: return {"status": "timeout", "input": len(toks), "output": 0, "response": ""} candidates = [k for k in CONCEPTS if k[:5] in toks] # concepts named in the prompt if len(candidates) < 3: candidates = CONCEPTS p = {k: 0.15 for k in candidates} for t in toks: for table, w in ((CUE, 1.0), (TRAP, 1.3)): if table.get(t) in p: p[table[t]] += w if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) # most likely choice else: e = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [v / sum(e) for v in e])] y = k if "tek" in toks else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k) out = tokenize(y) if len(out) > max_out: # stop condition out = out[:max_out] y = " ".join(out) return {"status": "done", "input": len(toks), "output": len(out), "response": y} def read_label(y): # extracting the label from the response for s in y.lower().split(): if s in TOPICS: return s return None THREE = { "plain": ("Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife , " "kesinti . Yanit olarak tek etiket yaz . Talep: "), "role": ("Sen destek masasi gorevlisisin . Talebi fatura , ariza , sayac , tarife , " "kesinti etiketlerinden birine ata , tek etiket yaz . Talep: "), "area": ("Once talebin hangi islem alanina girdigini belirle , sonra fatura , " "ariza , sayac , tarife , kesinti icinden tek etiket yaz . Talep: "), } def run_prompt(instruction, seed=SEED): COUNTER[0] = 0 labels, tin, tout, failed = [], 0, 0, 0 for x in TICKETS: s = call(instruction + x["text"], seed=seed) tin, tout, failed = tin + s["input"], tout + s["output"], failed + (s["status"] != "done") labels.append(read_label(s["response"])) return {"labels": labels, "input": tin, "output": tout, "calls": COUNTER[0], "failed": failed} def accuracy(labels): return sum(e == x["topic"] for e, x in zip(labels, TICKETS)) / len(TICKETS) RUN1 = {name: run_prompt(y) for name, y in THREE.items()} RUN2 = {name: run_prompt(y, SEED + 1) for name, y in THREE.items()} print(f"{'prompt':<7} {'tokens':>8} {'run 1':>8} {'run 2':>8} {'diff':>8} " f"{'input':>7} {'output':>6}") for name in THREE: a, b = accuracy(RUN1[name]["labels"]), accuracy(RUN2[name]["labels"]) print(f"{name:<7} {len(tokenize(THREE[name])):>8d} {a:>8.4f} {b:>8.4f} {b - a:>+8.4f} " f"{RUN1[name]['input']:>7d} {RUN1[name]['output']:>6d}") i = next(i for i in range(len(TICKETS)) if len({RUN1[name]["labels"][i] for name in THREE}) == 3) print(f"\nticket {TICKETS[i]['id']} ({TICKETS[i]['topic']}): {TICKETS[i]['text']}") print("labels of the three prompts: " + " , ".join(f"{name} -> {RUN1[name]['labels'][i]}" for name in THREE)) ``` ``` prompt tokens run 1 run 2 diff input output plain 32 0.7750 0.6917 -0.0833 5300 192 role 37 0.6583 0.7917 +0.1333 5900 210 area 34 0.7833 0.7500 -0.0333 5540 201 ticket 4023 (fatura): bu ayki fatura tutari gecen aya gore cok yuksek geldi bu konuda memnun degilim labels of the three prompts: plain -> fatura , role -> tarife , area -> sayac ``` Three prompts gave three separate numbers; the largest difference between them is **0.1250**. The real information is in the second run column. In the first run the best prompt is `area` (0.7833), the worst is `role` (0.6583); in the second run the ranking reverses and `role` takes the lead with **0.7917**. The difference of the same prompt between two runs is **0.0833**, **0.1333** and **0.0333**: the difference between prompts and the wobble of the same prompt within itself are **of the same size**. At this scale "the best prompt" is not a stable concept. The last two rows are the hope of the ensemble. Ticket 4023 is on the topic bill, and the three prompts give three separate labels; one of them knew it right. The ensemble's whole claim is this observation — if the errors fall in different places, vote combining eliminates them. ## Overlap of Errors and the Ceiling ```python from collections import Counter def vote(row): # majority; if all three labels differ, the first prompt decides s = Counter(e for e in row if e is not None) if not s: return None top = max(s.values()) return next((e for e in row if e is not None and s[e] == top), None) def analyze(RUN): rows = [[RUN[name]["labels"][i] for name in THREE] for i in range(len(TICKETS))] hit = [[row[j] == TICKETS[i]["topic"] for j in range(3)] for i, row in enumerate(rows)] ens = [vote(row) for row in rows] return {"ens": ens, "all_wrong": sum(not any(x) for x in hit), "ceiling": sum(any(x) for x in hit) / len(TICKETS), "no_consensus": sum(len(set(row)) == 3 for row in rows), "wrong_consensus": sum(1 for i, row in enumerate(rows) if Counter(row).most_common(1)[0][1] >= 2 and ens[i] != TICKETS[i]["topic"]), "accuracy": accuracy(ens), "calls": sum(RUN[n]["calls"] for n in THREE), "tokens": sum(RUN[n]["input"] + RUN[n]["output"] for n in THREE)} C1, C2 = analyze(RUN1), analyze(RUN2) best = max(accuracy(RUN1[n]["labels"]) for n in THREE) print(f"{'measure':<28} {'run 1':>9} {'run 2':>9}") for name, k, fmt in (("all three wrong", "all_wrong", "{:d}"), ("at least one right (ceiling)", "ceiling", "{:.4f}"), ("three different labels", "no_consensus", "{:d}"), ("agreement on a wrong label", "wrong_consensus", "{:d}"), ("ensemble accuracy", "accuracy", "{:.4f}")): print(f"{name:<28} {fmt.format(C1[k]):>9} {fmt.format(C2[k]):>9}") print(f"\nbest single prompt {best:.4f} -> ensemble {C1['accuracy']:.4f} " f"({C1['accuracy'] - best:+.4f}); distance to ceiling " f"{C1['accuracy'] - C1['ceiling']:+.4f}") single = RUN1["area"]["input"] + RUN1["area"]["output"] print(f"cost: calls {RUN1['area']['calls']} -> {C1['calls']} " f"({C1['calls'] / RUN1['area']['calls']:.1f} times), tokens {single} -> " f"{C1['tokens']} ({C1['tokens'] / single:.2f} times)") print(f"per correct label {single / (best * 120):.1f} -> " f"{C1['tokens'] / (C1['accuracy'] * 120):.1f} tokens") print("\nlabel changes between runs (over 120 records):") for name in THREE: print(f" {name:<9} {sum(a != b for a, b in zip(RUN1[name]['labels'], RUN2[name]['labels'])):>3d}") print(f" {'ensemble':<9} {sum(a != b for a, b in zip(C1['ens'], C2['ens'])):>3d}") ``` ``` measure run 1 run 2 all three wrong 9 16 at least one right (ceiling) 0.9250 0.8667 three different labels 1 8 agreement on a wrong label 24 18 ensemble accuracy 0.8000 0.8000 best single prompt 0.7833 -> ensemble 0.8000 (+0.0167); distance to ceiling -0.1250 cost: calls 120 -> 360 (3.0 times), tokens 5741 -> 17343 (3.02 times) per correct label 61.1 -> 180.7 tokens label changes between runs (over 120 records): plain 39 role 43 area 40 ensemble 24 ``` The first row answers this lesson's question directly. On **9** of the 120 records all three prompts are wrong at once; the overlap is **0.0750**. So the highest number the ensemble can reach is **0.9250**, and a gap of **0.1417** stands above the best single prompt. Overlap is low, the ceiling is high: up to here the ensemble's rationale stands. The realized number is **0.8000**, **0.1250** below the ceiling. The gap sits in a place that can be counted: the **agreement on a wrong label** row says the majority met on a wrong label in 24 records. Nine of these are records nobody could know anyway; on the remaining **15 records one prompt knew the right answer and lost the vote**. In a three-member ensemble the single member that knows is always in the minority. Overlap determines the ceiling, the vote determines how much of that ceiling will be reached; the two are separate numbers, and a report that writes only the first overstates the ensemble's gain by two times. The second run column shows how much the measure itself wobbles: only the seed changed, the overlap rose from 9 to **16**, the ceiling fell to **0.8667**. Overlap is a quantity that cannot be read from a single run. ## What Three Times the Calls Bought The gain is **+0.0167** against the best single prompt. The difference of the same prompts between two runs was between 0.0333 and 0.1333. **The gain is only half of even the smallest of the run differences; by this curriculum's rule it counts as unmeasured.** In the second run the gain is **+0.0083**; what its sign will be in the next run is not known. The cost, on the other hand, is measured and certain. The number of calls rose from 120 to **360** — exactly **three times**, because the number of members is directly the multiplier. The total tokens rose **3.02 times**, and the crumb in accuracy does not close this multiplier: the cost per correct label goes **from 61.1 tokens to 180.7 tokens**. A scaffolding that is three times as expensive giving a gain that cannot be measured is the exact counterpart of this curriculum's second claim — **reasoning techniques are expensive and do not pay off on every task.** The last table shows the one thing actually received in return for three calls. The single prompts change the labels of 39, 43 and 40 records between two runs; the ensemble changes them on **24 records**. What the vote does is not raise accuracy but **stabilize the output**, and it leaves that half done. The accuracy of the two runs came out **0.8000** and **0.8000**; under these two equal numbers there are 24 labels that changed. Assuming it is stable because the totals are equal is the very mistake this curriculum warns against. A prompt ensemble pays off when the members' errors really fall in different places. Here all three instructions carried the same five labels and the same format constraint; their differences stayed in the wording, not in the decision. Saying the same information in three separate sentences does not produce three viewpoints, it produces three bills. ## Summary - Three separate instructions give **0.7750**, **0.6583** and **0.7833** on the same task; in the second run the ranking reverses. The difference between prompts and the run difference are of the same size. - Overlap is counted: on **9** of the 120 records all three are wrong at once, the ceiling is **0.9250**. In the second run these numbers are **16** and **0.8667**. - The majority vote stays at **0.8000** — **0.1250** below the ceiling. On **15 records** one prompt knew the right answer and lost the vote; the majority met on a wrong label in 24 records. - The gain is **+0.0167** against the best single prompt, and the two-run difference is between 0.0333 and 0.1333. The gain counts as unmeasured. - The cost is written in number of calls: 120 calls **360 calls**, 61.1 tokens per correct label **180.7 tokens**. The only thing received was the change between runs falling from 40 to **24**. ## Next Step In this lesson three prompts were written by hand and which one was better changed with the run. There was a criterion, a set and candidate prompts at hand; only the choice was made by eye and by looking at a single run. Turning the same job into a loop is possible: candidates are produced, all are scored with the same criterion and the highest is chosen. The next lesson builds this loop and asks a single question — as the search budget grows, how much of the number that rises on the validation set carries over to a test set that has never been seen.