Self-Consistency

The previous lesson showed that asking for a step-by-step solution works on the multi-step calculation: accuracy rose from 0.3000 to 0.8833. But that number came from a single run. The model is an uncertain component; the same prompt produces another chain on a second run, and if the chain slips somewhere the result changes. With a single chain in hand, nothing can say whether it is right.

Self-consistency turns this uncertainty into a lever: the same prompt is run several times and the responses that come out are voted on. In this lesson the text of the prompt never changes; the only thing that changes is how many times the same prompt is run. This is why the cost is written with the number of calls alongside the token. Majority voting itself was measured in the ensemble methods lesson of the previous curriculum and is not repeated here; what is added is the cost and the place where the consensus breaks.

• RT11. The code does not call a real endpoint; it calls a language model simulator written with the standard library. Seed 20260218. This lesson uses only the calculation path.
• RT12. The task is the previous lesson’s: a 60-record fictional tariff calculation, tiers of 10, 25 and 40 m³, with a fixed fee and a subscriber type discount on top. The criterion is exact equality.
• RT13. The prompt is the previous lesson’s stepwise prompt and stays the same letter for letter.
• RT14. On a real endpoint the diversity among samples comes from a distribution temperature above zero; in the simulator the seed is taken one higher for each sample.
• RT15. The numbers of samples are 1, 3, 5, 7 and 9; it does not go beyond nine.
• RT16. In the vote the answer with the most votes is chosen, and on a tie the answer that got there first wins. A call that hits the rate limit cannot vote, but it enters the number of calls and the token cost.
• RT17. Consensus: there is consensus if more than half of the valid votes are on the same answer. With a single sample the consensus always forms and says nothing.
• RT18. The simulator’s step slip is of two kinds: a 0.05 scattered slip at every step, and a 0.85 systematic slip if the remainder falling on the last tier is 5 m³ or less. The second is a modeling assumption and is the source of what this lesson measures.
• RT19. The whole measurement is repeated with two seed bases: 20260218 and 20260258.
• RT20. The cost is three numbers: total tokens, the number of calls and the tokens per correct response.

Nine Samples of the Same Prompt

First, what happens on a single record is examined. The same prompt is run nine times, the seed goes up by one in each run, and the last number in the response counts as one vote. Two records were chosen: one ordinary, the other just above the 40 m³ limit.

# SIMULATOR. This code does not call a real model endpoint; it is a language model
# simulator written with the standard library. The tickets and the tariff are FICTIONAL too.
SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW, MAX_OUT_STEPS = 240, 96     # context window and the output limit for the stepwise response


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(text):                 # FICTIONAL tokenizer: a piece has at most five characters
    out = []
    for piece in text.lower().replace(",", " , ").split():
        while len(piece) > 5:
            out.append(piece[:5])
            piece = piece[5:]
        out.append(piece)
    return out


def trace_id(text):                 # depends on the prompt's content, independent of the run
    h = 2166136261
    for c in text:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


# FICTIONAL tariff: tier upper bound (m3) and unit price, fixed fee, subscriber type discount
TIERS, FIXED_FEE = [(10, 3), (25, 5), (40, 8), (10 ** 6, 12)], 20
SUBSCRIBER_TYPES = {"konut": 0, "tarim": 20, "resmi": 10}
CASES = []
for i in range(60):
    r = rng(SEED + 900 + i)
    m3 = 6 + int(r() * 52)
    sub_type = sorted(SUBSCRIBER_TYPES)[pick(r(), [0.40, 0.34, 0.26])]
    CASES.append({"id": 7001 + i, "m3": m3, "type": sub_type,
                  "text": f"tuketim {m3} m3 abone tipi {sub_type} tutar ne kadar"})


def tiered_total(m3):               # total of the consumption split across tiers
    low, left, total = 0, m3, 0
    for high, unit in TIERS:
        share = min(left, high - low)
        total, left, low = total + share * unit, left - share, high
        if left <= 0:
            break
    return total


def exact_answer(m3, sub_type):     # the task's correct answer, outside the simulator
    return ((tiered_total(m3) + FIXED_FEE) * (100 - SUBSCRIBER_TYPES[sub_type])) // 100


COUNTER = [0]


def _parse_task(toks):              # consumption and subscriber type from the ticket at the end of the prompt
    return (next((int(u) for u in reversed(toks) if u.isdigit()), 0),
            next((t for t in reversed(toks) if t in SUBSCRIBER_TYPES), "konut"))


def _step_by_step(m3, sub_type, r):  # each tier written separately, a slip margin per step
    low, left, total, steps = 0, m3, 0, []
    for i, (high, unit) in enumerate(TIERS):
        share = min(left, high - low)
        # if the remainder in the last tier is small the slip is systematic: it always goes to the same wrong value
        u = unit
        if r() < (0.85 if i == 3 and share <= 5 else 0.05):
            u = TIERS[i - 1][1] if i + 1 == len(TIERS) else TIERS[i + 1][1]
        total, left, low = total + share * u, left - share, high
        steps.append(f"basamak {i + 1} , {share} m3 , birim {u} , ara toplam {total}")
        if left <= 0:
            break
    if r() < 0.05:
        steps.append(f"sabit bedel atlandi ara toplam {total}")
    else:
        total += FIXED_FEE
        steps.append(f"sabit bedel {FIXED_FEE} ara toplam {total}")
    discount = 0 if r() < 0.05 else SUBSCRIBER_TYPES[sub_type]
    total = (total * (100 - discount)) // 100
    return " . ".join(steps + [f"indirim {discount} sonuc {total}"])


def call(prompt, seed=SEED, max_out=MAX_OUT_STEPS):
    COUNTER[0] += 1
    toks = tokenize(prompt)
    if len(toks) > WINDOW:
        return {"status": "context_overflow", "input": len(toks), "output": 0, "response": ""}
    if COUNTER[0] % 37 == 0:                    # the rate limit depends on the number of calls
        return {"status": "rate_limit", "input": len(toks), "output": 0, "response": ""}
    m3, sub_type = _parse_task(toks)
    y = _step_by_step(m3, sub_type, rng(seed ^ trace_id(prompt)))
    out = tokenize(y)
    if len(out) > max_out:                      # stop condition
        out = out[:max_out]
        y = " ".join(out)
    return {"status": "ok", "input": len(toks), "output": len(out), "response": y}


def read_number(y):                             # the last number in the response
    return next((int(x) for x in reversed(y.split()) if x.isdigit()), None)


PROMPT = ("Tarife: ilk 10 m3 birim 3 , sonraki 15 m3 birim 5 , sonraki 15 m3 birim 8 , "
          "40 ustu birim 12 . Sabit bedel 20 . Abone tipi indirimi yuzde konut 0 , "
          "tarim 20 , resmi 10 . Tutari hesapla . Once her basamagi ayri ayri adim adim "
          "yaz , sonra sonucu yaz . Talep: ")


def sample_vote(x, n, base):        # the same prompt is run n times, the responses are voted on
    votes = {}
    for j in range(n):
        v = read_number(call(PROMPT + x["text"], seed=base + j)["response"])
        if v is not None:                       # a call that hits the rate limit does not vote
            votes[v] = votes.get(v, 0) + 1
    if not votes:
        return None, 0
    top = max(votes.values())
    return next(k for k in votes if votes[k] == top), top
COUNTER[0] = 0
for x in (CASES[3], next(c for c in CASES if 40 < c["m3"] <= 45)):
    votes, correct = {}, exact_answer(x["m3"], x["type"])
    for j in range(9):
        v = read_number(call(PROMPT + x["text"], seed=SEED + j)["response"])
        votes[v] = votes.get(v, 0) + 1
    chosen, top = sample_vote(x, 9, SEED)
    print(f"record {x['id']}: {x['m3']} m3 , {x['type']} , correct answer {correct}")
    print(f"  nine samples: {sorted(votes.items(), key=lambda p: -p[1])}")
    print(f"  consensus {chosen} ({top}/9) -> {'correct' if chosen == correct else 'WRONG'}")

record 7004: 12 m3 , resmi , correct answer 54
  nine samples: [(54, 6), (36, 2), (59, 1)]
  consensus 54 (6/9) -> correct
record 7008: 45 m3 , tarim , correct answer 244
  nine samples: [(228, 5), (276, 2), (264, 1), (244, 1)]
  consensus 228 (5/9) -> WRONG

On the first record voting works. Six of the nine samples say 54, and that is the correct answer. The remaining three votes are spread over two different wrong answers: 36, the response where the fixed fee was skipped; 59, the response where the unit of the second tier slipped. The errors differ from one another, while the correct answer is the same in all of them. This is why scattered error is fixed by voting — there are many ways to go wrong, and one way to go right.

On the second record the same scaffolding works in reverse. The correct answer is 244, and only one of the nine samples says it. The majority, five of the nine, gathers on 228: the last 5 m³ of the 45 m³ was calculated with the unit of one tier lower rather than the unit of the top tier. Voting does not fix this record, it reinforces the error. The difference lies in the type of the error: the first is scattered, the second systematic.

Number of Samples, Gain and Calls

The same measurement is repeated on the whole set and at five different numbers of samples. Each row is run with two seed bases; the second column is the second run of the same scaffolding.

def measure(n, base):
    COUNTER[0] = 0
    n_in = n_out = correct = agreed = empty = 0
    broken = []
    for x in CASES:
        votes, answer = {}, exact_answer(x["m3"], x["type"])
        for j in range(n):
            s = call(PROMPT + x["text"], seed=base + j)
            n_in, n_out = n_in + s["input"], n_out + s["output"]
            v = read_number(s["response"])
            if v is None:
                empty += 1                      # a call that hits the rate limit does not vote
            else:
                votes[v] = votes.get(v, 0) + 1
        if not votes:
            continue                            # the single sample hit the rate limit
        top = max(votes.values())
        chosen = next(k for k in votes if votes[k] == top)
        correct += chosen == answer
        if top * 2 > sum(votes.values()):       # more than half on the same answer
            agreed += 1
            if chosen != answer:
                broken.append(x["m3"])
    return {"accuracy": correct / len(CASES), "calls": COUNTER[0], "tokens": n_in + n_out,
            "agreed": agreed, "broken": broken, "empty": empty}


print(f"{'samples':>7} {'accuracy':>8} {'second':>8} {'calls':>6} {'tokens':>9} "
      f"{'per_correct':>11} {'agreed':>8} {'on_wrong':>9}")
for n in (1, 3, 5, 7, 9):
    A, B = measure(n, SEED), measure(n, SEED + 40)
    print(f"{n:>7} {A['accuracy']:>8.4f} {B['accuracy']:>8.4f} {A['calls']:>6} "
          f"{A['tokens']:>9} {A['tokens'] / A['accuracy'] / len(CASES):>11.1f} "
          f"{A['agreed']:>8} {len(A['broken']):>9}")
    if n == 9:
        print(f"\nconsumption of the records with a wrong consensus at nine samples: "
              f"{sorted(A['broken'], reverse=True)} m3")
        print(f"calls that hit the rate limit and could not vote: {A['empty']}")

samples accuracy   second  calls    tokens per_correct   agreed  on_wrong
      1   0.7667   0.6667     60      7954       172.9       59        13
      3   0.8500   0.8667    180     23785       466.4       56         6
      5   0.8667   0.8833    300     39552       760.6       55         4
      7   0.8833   0.9000    420     55389      1045.1       57         5
      9   0.8833   0.8833    540     71211      1343.6       58         5

consumption of the records with a wrong consensus at nine samples: [45, 45, 44, 43, 41] m3
calls that hit the rate limit and could not vote: 14

The first column gives the gain. With one sample accuracy is 0.7667, with nine samples 0.8833; the gain is +0.1167. The distribution of the gain is not even: the first three samples bring +0.0833, the next four samples only +0.0333, and the move from seven to nine brings nothing. The gain saturates at the seventh sample.

The second column is the most important number of this lesson. With one sample the two runs give 0.7667 and 0.6667; the difference between them is 0.1000. With nine samples both runs give 0.8833; the difference is 0.0000. The one-sample gain (+0.1167) stays only a little above the run difference of one sample (0.1000), so it would be hard to count it as measured by looking at the first run. On the second run the same gain is +0.2166. The most robustly measured effect of the technique is not that it raises accuracy, but that it brings the wobble from run to run down to zero.

The third and fourth columns are the cost, and it is not debatable. The number of calls goes from 60 to 540, total tokens from 7954 to 71211 — almost exactly nine times. The cost per correct response rises from 172.9 to 1343.6 tokens, that is, 7.8 times. A cost nine times larger while the gain stays at 0.1167 says in a single line why this technique cannot be switched on for every task. The number for three samples (0.8500, 466.4 tokens) is in most cases a more defensible point than nine samples.

Where the Consensus Breaks

The last two columns look at the voting itself. With one sample there is consensus on 59 records and 13 of them are wrong, because a single vote is always the majority. This shows that consensus alone says nothing. With nine samples the records with consensus are 58, and the ones with consensus on a wrong answer 5. The number of records with consensus on a wrong answer, as the number of samples goes from three to nine, gets stuck at 6, 4, 5 and 5.

The consumption of these five records is printed: 45, 45, 44, 43 and 41 m³. All of them are just above the 40 m³ limit. In these records the remainder falling on the last tier is small, and the simulator calculates it systematically from the unit of one tier lower. The error is not random, which is why all nine samples go to the same wrong answer. Self-consistency fixes scattered error and reinforces systematic error. Consensus is not proof of correctness; it is only a sign that the error is not scattered.

There is one more place where the cost does not turn into gain: 14 of the 540 calls hit the rate limit, could not vote, and still entered the number of calls. Every scaffolding that multiplies the number of calls by nine also multiplies the surface’s failure share by nine.

Summary

• Self-consistency means running the same prompt several times and voting on the responses; the text of the prompt does not change, the number of runs does.
• As the number of samples goes from one to nine, accuracy rises from 0.7667 to 0.8833; +0.0833 of the gain comes from the first three samples and the gain stops after the seventh sample.
• The cost is written in two units: calls from 60 to 540, total tokens from 7954 to 71211, tokens per correct response from 172.9 to 1343.6. Tokens nine times, gain 0.1167.
• The most robustly measured effect of the technique is that it brings the run difference down from 0.1000 to 0.0000; the gain of a single sample stays only a little above the run difference of a single sample.
• With nine samples a consensus forms on 58 records, and on 5 of them the consensus is on a wrong answer. The consumption of these five records is between 41 and 45 m³: when the error is systematic, voting does not fix it, it reinforces it.

Next Step

Generating several solutions and voting worked, but the generated solutions had one thing in common: each was built independently from start to finish. Nine samples started from zero nine times, none saw where the other slipped, and none took over the good first half of another. Branching from the middle of a solution was never tried. The next lesson tries it: the solution stops being a single chain and becomes a structure that grows step by step and splits in several directions at every step. What will be measured is the branching factor and how the number of calls grows as the depth increases, and where the gain stops.