--- title: Self-Consistency source: 'https://academia.sh/en/courses/prompt-engineering/self-consistency' course: 'Prompt Engineering' language: en updated: '2026-10-21T05:02:34+00:00' license: 'CC BY-SA 4.0' objectives: - 'Builds a scaffolding that runs the same prompt several times and votes on the responses, and measures where the gain saturates with the number of samples.' - 'Writes the cost of the technique in two units: total tokens and the number of calls; calculates the tokens per correct response by the number of samples.' - 'Counts the records where the consensus forms on a wrong answer and tells scattered error apart from systematic error.' --- # Self-Consistency The gain and the cost of running the same prompt several times and voting on the responses: as the number of samples goes from one to nine, accuracy rises from 0.7667 to 0.8833 and stops after seven, the number of calls goes from 60 to 540 and the tokens per correct response from 172.9 to 1343.6. At nine samples a consensus forms on 58 records, but on five of them the consensus is on a wrong answer; the consumption of those five records lies between 41 and 45 m³, that is, where the error is systematic rather than scattered. The previous lesson showed that asking for a step-by-step solution works on the multi-step calculation: accuracy rose from 0.3000 to 0.8833. But that number came from **a single run**. The model is an uncertain component; the same prompt produces another chain on a second run, and if the chain slips somewhere the result changes. With a single chain in hand, nothing can say whether it is right. Self-consistency turns this uncertainty into a lever: the same prompt is run several times and the responses that come out are **voted on**. In this lesson **the text of the prompt never changes**; the only thing that changes is how many times the same prompt is run. This is why the cost is written with the **number of calls** alongside the token. Majority voting itself was measured in the ensemble methods lesson of the previous curriculum and is not repeated here; what is added is the cost and **the place where the consensus breaks**. - **RT11.** The code does not call a real endpoint; it calls a language model **simulator** written with the standard library. Seed **20260218**. This lesson uses only the calculation path. - **RT12.** The task is the previous lesson's: a 60-record **fictional** tariff calculation, tiers of 10, 25 and 40 m³, with a fixed fee and a subscriber type discount on top. The criterion is **exact equality**. - **RT13.** The prompt is the previous lesson's stepwise prompt and stays the same letter for letter. - **RT14.** On a real endpoint the diversity among samples comes from a **distribution temperature** above zero; in the simulator the seed is taken one higher for each sample. - **RT15.** The numbers of samples are 1, 3, 5, 7 and 9; it **does not go beyond nine**. - **RT16.** In the vote the answer with the most votes is chosen, and on a tie the answer that got there first wins. **A call that hits the rate limit cannot vote**, but it enters the number of calls and the token cost. - **RT17.** **Consensus**: there is consensus if more than half of the valid votes are on the same answer. With a single sample the consensus always forms and says nothing. - **RT18.** The simulator's step slip is of two kinds: a **0.05 scattered slip** at every step, and a **0.85 systematic slip** if the remainder falling on the last tier is 5 m³ or less. The second is a modeling assumption and is the source of what this lesson measures. - **RT19.** The whole measurement is repeated with two seed bases: **20260218** and **20260258**. - **RT20.** The cost is three numbers: total tokens, the **number of calls** and the tokens per correct response. ## Nine Samples of the Same Prompt First, what happens on a single record is examined. The same prompt is run nine times, the seed goes up by one in each run, and the last number in the response counts as one vote. Two records were chosen: one ordinary, the other just above the 40 m³ limit. ```python # SIMULATOR. This code does not call a real model endpoint; it is a language model # simulator written with the standard library. The tickets and the tariff are FICTIONAL too. SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUT_STEPS = 240, 96 # context window and the output limit for the stepwise response def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): # FICTIONAL tokenizer: a piece has at most five characters out = [] for piece in text.lower().replace(",", " , ").split(): while len(piece) > 5: out.append(piece[:5]) piece = piece[5:] out.append(piece) return out def trace_id(text): # depends on the prompt's content, independent of the run h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h # FICTIONAL tariff: tier upper bound (m3) and unit price, fixed fee, subscriber type discount TIERS, FIXED_FEE = [(10, 3), (25, 5), (40, 8), (10 ** 6, 12)], 20 SUBSCRIBER_TYPES = {"konut": 0, "tarim": 20, "resmi": 10} CASES = [] for i in range(60): r = rng(SEED + 900 + i) m3 = 6 + int(r() * 52) sub_type = sorted(SUBSCRIBER_TYPES)[pick(r(), [0.40, 0.34, 0.26])] CASES.append({"id": 7001 + i, "m3": m3, "type": sub_type, "text": f"tuketim {m3} m3 abone tipi {sub_type} tutar ne kadar"}) def tiered_total(m3): # total of the consumption split across tiers low, left, total = 0, m3, 0 for high, unit in TIERS: share = min(left, high - low) total, left, low = total + share * unit, left - share, high if left <= 0: break return total def exact_answer(m3, sub_type): # the task's correct answer, outside the simulator return ((tiered_total(m3) + FIXED_FEE) * (100 - SUBSCRIBER_TYPES[sub_type])) // 100 COUNTER = [0] def _parse_task(toks): # consumption and subscriber type from the ticket at the end of the prompt return (next((int(u) for u in reversed(toks) if u.isdigit()), 0), next((t for t in reversed(toks) if t in SUBSCRIBER_TYPES), "konut")) def _step_by_step(m3, sub_type, r): # each tier written separately, a slip margin per step low, left, total, steps = 0, m3, 0, [] for i, (high, unit) in enumerate(TIERS): share = min(left, high - low) # if the remainder in the last tier is small the slip is systematic: it always goes to the same wrong value u = unit if r() < (0.85 if i == 3 and share <= 5 else 0.05): u = TIERS[i - 1][1] if i + 1 == len(TIERS) else TIERS[i + 1][1] total, left, low = total + share * u, left - share, high steps.append(f"basamak {i + 1} , {share} m3 , birim {u} , ara toplam {total}") if left <= 0: break if r() < 0.05: steps.append(f"sabit bedel atlandi ara toplam {total}") else: total += FIXED_FEE steps.append(f"sabit bedel {FIXED_FEE} ara toplam {total}") discount = 0 if r() < 0.05 else SUBSCRIBER_TYPES[sub_type] total = (total * (100 - discount)) // 100 return " . ".join(steps + [f"indirim {discount} sonuc {total}"]) def call(prompt, seed=SEED, max_out=MAX_OUT_STEPS): COUNTER[0] += 1 toks = tokenize(prompt) if len(toks) > WINDOW: return {"status": "context_overflow", "input": len(toks), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: # the rate limit depends on the number of calls return {"status": "rate_limit", "input": len(toks), "output": 0, "response": ""} m3, sub_type = _parse_task(toks) y = _step_by_step(m3, sub_type, rng(seed ^ trace_id(prompt))) out = tokenize(y) if len(out) > max_out: # stop condition out = out[:max_out] y = " ".join(out) return {"status": "ok", "input": len(toks), "output": len(out), "response": y} def read_number(y): # the last number in the response return next((int(x) for x in reversed(y.split()) if x.isdigit()), None) PROMPT = ("Tarife: ilk 10 m3 birim 3 , sonraki 15 m3 birim 5 , sonraki 15 m3 birim 8 , " "40 ustu birim 12 . Sabit bedel 20 . Abone tipi indirimi yuzde konut 0 , " "tarim 20 , resmi 10 . Tutari hesapla . Once her basamagi ayri ayri adim adim " "yaz , sonra sonucu yaz . Talep: ") def sample_vote(x, n, base): # the same prompt is run n times, the responses are voted on votes = {} for j in range(n): v = read_number(call(PROMPT + x["text"], seed=base + j)["response"]) if v is not None: # a call that hits the rate limit does not vote votes[v] = votes.get(v, 0) + 1 if not votes: return None, 0 top = max(votes.values()) return next(k for k in votes if votes[k] == top), top COUNTER[0] = 0 for x in (CASES[3], next(c for c in CASES if 40 < c["m3"] <= 45)): votes, correct = {}, exact_answer(x["m3"], x["type"]) for j in range(9): v = read_number(call(PROMPT + x["text"], seed=SEED + j)["response"]) votes[v] = votes.get(v, 0) + 1 chosen, top = sample_vote(x, 9, SEED) print(f"record {x['id']}: {x['m3']} m3 , {x['type']} , correct answer {correct}") print(f" nine samples: {sorted(votes.items(), key=lambda p: -p[1])}") print(f" consensus {chosen} ({top}/9) -> {'correct' if chosen == correct else 'WRONG'}") ``` ``` record 7004: 12 m3 , resmi , correct answer 54 nine samples: [(54, 6), (36, 2), (59, 1)] consensus 54 (6/9) -> correct record 7008: 45 m3 , tarim , correct answer 244 nine samples: [(228, 5), (276, 2), (264, 1), (244, 1)] consensus 228 (5/9) -> WRONG ``` On the first record voting works. Six of the nine samples say **54**, and that is the correct answer. The remaining three votes are spread over two different wrong answers: **36**, the response where the fixed fee was skipped; **59**, the response where the unit of the second tier slipped. The errors differ from one another, while the correct answer is the same in all of them. This is why scattered error is fixed by voting — there are many ways to go wrong, and one way to go right. On the second record the same scaffolding works in reverse. The correct answer is **244**, and only **one** of the nine samples says it. The majority, **five** of the nine, gathers on **228**: the last 5 m³ of the 45 m³ was calculated with the unit of one tier lower rather than the unit of the top tier. Voting does not fix this record, it **reinforces the error**. The difference lies in the type of the error: the first is scattered, the second systematic. ## Number of Samples, Gain and Calls The same measurement is repeated on the whole set and at five different numbers of samples. Each row is run with two seed bases; the second column is the second run of the same scaffolding. ```python def measure(n, base): COUNTER[0] = 0 n_in = n_out = correct = agreed = empty = 0 broken = [] for x in CASES: votes, answer = {}, exact_answer(x["m3"], x["type"]) for j in range(n): s = call(PROMPT + x["text"], seed=base + j) n_in, n_out = n_in + s["input"], n_out + s["output"] v = read_number(s["response"]) if v is None: empty += 1 # a call that hits the rate limit does not vote else: votes[v] = votes.get(v, 0) + 1 if not votes: continue # the single sample hit the rate limit top = max(votes.values()) chosen = next(k for k in votes if votes[k] == top) correct += chosen == answer if top * 2 > sum(votes.values()): # more than half on the same answer agreed += 1 if chosen != answer: broken.append(x["m3"]) return {"accuracy": correct / len(CASES), "calls": COUNTER[0], "tokens": n_in + n_out, "agreed": agreed, "broken": broken, "empty": empty} print(f"{'samples':>7} {'accuracy':>8} {'second':>8} {'calls':>6} {'tokens':>9} " f"{'per_correct':>11} {'agreed':>8} {'on_wrong':>9}") for n in (1, 3, 5, 7, 9): A, B = measure(n, SEED), measure(n, SEED + 40) print(f"{n:>7} {A['accuracy']:>8.4f} {B['accuracy']:>8.4f} {A['calls']:>6} " f"{A['tokens']:>9} {A['tokens'] / A['accuracy'] / len(CASES):>11.1f} " f"{A['agreed']:>8} {len(A['broken']):>9}") if n == 9: print(f"\nconsumption of the records with a wrong consensus at nine samples: " f"{sorted(A['broken'], reverse=True)} m3") print(f"calls that hit the rate limit and could not vote: {A['empty']}") ``` ``` samples accuracy second calls tokens per_correct agreed on_wrong 1 0.7667 0.6667 60 7954 172.9 59 13 3 0.8500 0.8667 180 23785 466.4 56 6 5 0.8667 0.8833 300 39552 760.6 55 4 7 0.8833 0.9000 420 55389 1045.1 57 5 9 0.8833 0.8833 540 71211 1343.6 58 5 consumption of the records with a wrong consensus at nine samples: [45, 45, 44, 43, 41] m3 calls that hit the rate limit and could not vote: 14 ``` The first column gives the gain. With one sample accuracy is **0.7667**, with nine samples **0.8833**; the gain is **+0.1167**. The distribution of the gain is not even: the first three samples bring **+0.0833**, the next four samples only **+0.0333**, and the move from seven to nine brings **nothing**. The gain saturates at the seventh sample. The second column is the most important number of this lesson. With one sample the two runs give **0.7667** and **0.6667**; the difference between them is **0.1000**. With nine samples both runs give **0.8833**; the difference is **0.0000**. The one-sample gain (+0.1167) stays only a little above the run difference of one sample (0.1000), so it would be hard to count it as measured by looking at the first run. On the second run the same gain is **+0.2166**. The most robustly measured effect of the technique is not that it raises accuracy, but that it **brings the wobble from run to run down to zero**. The third and fourth columns are the cost, and it is not debatable. The number of calls goes **from 60 to 540**, total tokens **from 7954 to 71211** — almost exactly nine times. The cost per correct response rises **from 172.9 to 1343.6 tokens**, that is, **7.8 times**. A cost nine times larger while the gain stays at 0.1167 says in a single line why this technique cannot be switched on for every task. The number for three samples (0.8500, 466.4 tokens) is in most cases a more defensible point than nine samples. ## Where the Consensus Breaks The last two columns look at the voting itself. With one sample **there is consensus on 59 records** and **13 of them are wrong**, because a single vote is always the majority. This shows that consensus alone says nothing. With nine samples the records with consensus are **58**, and the ones with consensus on a wrong answer **5**. The number of records with consensus on a wrong answer, as the number of samples goes from three to nine, **gets stuck at 6, 4, 5 and 5**. The consumption of these five records is printed: **45, 45, 44, 43 and 41 m³**. All of them are just above the 40 m³ limit. In these records the remainder falling on the last tier is small, and the simulator calculates it systematically from the unit of one tier lower. The error is not random, which is why all nine samples go to the same wrong answer. **Self-consistency fixes scattered error and reinforces systematic error.** Consensus is not proof of correctness; it is only a sign that the error is not scattered. There is one more place where the cost does not turn into gain: **14** of the 540 calls hit the rate limit, could not vote, and still entered the number of calls. Every scaffolding that multiplies the number of calls by nine also multiplies the surface's failure share by nine. ## Summary - Self-consistency means running **the same prompt** several times and voting on the responses; the text of the prompt does not change, the number of runs does. - As the number of samples goes from one to nine, accuracy rises **from 0.7667 to 0.8833**; **+0.0833** of the gain comes **from the first three samples** and the gain stops after the seventh sample. - The cost is written in two units: calls **from 60 to 540**, total tokens **from 7954 to 71211**, tokens per correct response **from 172.9 to 1343.6**. Tokens nine times, gain 0.1167. - The most robustly measured effect of the technique is that it brings the run difference down **from 0.1000 to 0.0000**; the gain of a single sample stays only a little above the run difference of a single sample. - With nine samples a consensus forms on 58 records, and on **5 of them the consensus is on a wrong answer**. The consumption of these five records is between 41 and 45 m³: when the error is systematic, voting does not fix it, it reinforces it. ## Next Step Generating several solutions and voting worked, but the generated solutions had one thing in common: each was built **independently from start to finish**. Nine samples started from zero nine times, none saw where the other slipped, and none took over the good first half of another. **Branching from the middle** of a solution was never tried. The next lesson tries it: the solution stops being a single chain and becomes a structure that grows step by step and splits in several directions at every step. What will be measured is the branching factor and **how the number of calls grows** as the depth increases, and where the gain stops.