Chain of Thought

The prompt design topic spent eight lessons arranging what to say in a prompt: instruction, role, measurable request, format, example, negative constraint and iterating on all of them by measuring. None of them asked how the model should think. The prompt the eighth lesson left behind still asks a single thing and expects a single response.

Chain of thought fills this gap: asking the model not for the result but for the intermediate steps that lead to it. This lesson measures that one-sentence change on two tasks, because the second claim of the course is built here: reasoning techniques are expensive and do not pay off on every task.

• RT1. The code does not call a real model endpoint; it calls a language model simulator written with the standard library. Seed 20260218.
• RT2. The support tickets and the tariff are fictional: 120 short support tickets, 60 account records.
• RT3. The tiers of the fictional tariff are 10, 25 and 40 m³; on top of them come a fixed fee and a discount that depends on the subscriber type. The tariff description is inside the prompt and counts toward the token cost.
• RT4. The simulator has two calculation paths: the one-move path that picks one of the shortcut patterns, and the step-by-step path that calculates each tier separately and carries a 0.05 slip margin per step. This distinction is a modeling assumption.
• RT5. In classification, asking for step by step changes only the format of the response; the label scoring stays the same. The criterion is exact equality in the calculation and label accuracy in classification.
• RT6. The stepwise response is long; the maximum output length is raised from 24 to 96 tokens, otherwise the stop condition cuts the result off. This expansion is also part of the cost.
• RT7. Run band: the same prompt is run on three seeds, and the difference between the highest and the lowest accuracy is the band. A change whose difference stays inside the band counts as unmeasured.

Two Versions of the Same Prompt

Three numbers are fixed: the previous prompt, the changed prompt, the cost. The previous prompt is the version the prompt design topic left behind: task defined, tariff given, format imposed. The changed prompt adds a single sentence to it — first write each tier separately, step by step.

# SIMULATOR. This code does not call a real model endpoint; it is a language model
# simulator written with the standard library. The tickets and the tariff are FICTIONAL too.
import math

SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW, MAX_OUT, MAX_OUT_STEPS = 240, 24, 96     # context window and output limits


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(text):                 # FICTIONAL tokenizer: a piece has at most five characters
    out = []
    for piece in text.lower().replace(",", " , ").split():
        while len(piece) > 5:
            out.append(piece[:5])
            piece = piece[5:]
        out.append(piece)
    return out


def trace_id(text):                 # depends on the prompt's content, independent of the run
    h = 2166136261
    for c in text:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
PATTERNS = {
    "fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi",
               "faturami odedim ama sistemde borc gorunuyor",
               "fatura ayrintisini nereden gorebilirim"],
    "ariza": ["evde su akmiyor bir ariza var galiba",
              "baglanti bozuk oldu ve hic calismiyor",
              "ariza kaydi actim ama ekip gelmedi"],
    "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor",
              "sayac degisimi icin randevu istiyorum",
              "sayac uzerindeki endeks ile fatura ayni degil"],
    "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum",
               "tarife degisikligi icin ne yapmam gerekiyor",
               "birim ucret abone tipine gore degisiyor mu"],
    "kesinti": ["mahallede kesinti var ne zaman gelecek",
                "planli kesinti duyurusu nereden takip edilir",
                "aksamdan beri kesinti yasiyoruz"],
}
EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim",
          "acil bilgi almam gerekiyor", "kayit numaram elimde yok"]
TICKETS = []
for i in range(120):
    r = rng(SEED + i)
    topic = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    text = PATTERNS[topic][int(r() * 3)]
    if r() < 0.34:
        text += " " + EXTRAS[int(r() * 4)]
    TICKETS.append({"id": 4001 + i, "text": text, "topic": topic})

CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura",
        "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza",
        "sayac": "sayac", "endek": "sayac", "okuma": "sayac",
        "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife",
        "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti"}
TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura",
         "ekip": "kesinti", "takip": "tarife", "giril": "fatura"}   # misleading cues
RATIONALE = ("birinci adim : talep metnindeki isaretler tarandi . ikinci adim : en guclu "
             "isaret {topic} kavramini gosteriyor . ucuncu adim : etiket {topic}")

# FICTIONAL tariff: tier upper bound (m3) and unit price, fixed fee, subscriber type discount
TIERS, FIXED_FEE = [(10, 3), (25, 5), (40, 8), (10 ** 6, 12)], 20
SUBSCRIBER_TYPES = {"konut": 0, "tarim": 20, "resmi": 10}
CASES = []
for i in range(60):
    r = rng(SEED + 900 + i)
    m3 = 6 + int(r() * 52)
    sub_type = sorted(SUBSCRIBER_TYPES)[pick(r(), [0.40, 0.34, 0.26])]
    CASES.append({"id": 7001 + i, "m3": m3, "type": sub_type,
                  "text": f"tuketim {m3} m3 abone tipi {sub_type} tutar ne kadar"})


def tiered_total(m3):               # total of the consumption split across tiers
    low, left, total = 0, m3, 0
    for high, unit in TIERS:
        share = min(left, high - low)
        total, left, low = total + share * unit, left - share, high
        if left <= 0:
            break
    return total


def exact_answer(m3, sub_type):     # the task's correct answer, outside the simulator
    return ((tiered_total(m3) + FIXED_FEE) * (100 - SUBSCRIBER_TYPES[sub_type])) // 100


COUNTER = [0]


def _parse_task(toks):              # consumption and subscriber type from the ticket at the end
    return (next((int(u) for u in reversed(toks) if u.isdigit()), 0),
            next((t for t in reversed(toks) if t in SUBSCRIBER_TYPES), "konut"))


def _step_by_step(m3, sub_type, r, slip=0.05):  # each tier separate, a slip per step
    low, left, total, steps = 0, m3, 0, []
    for i, (high, unit) in enumerate(TIERS):
        share = min(left, high - low)
        u = TIERS[i + 1][1] if r() < slip and i + 1 < len(TIERS) else unit
        total, left, low = total + share * u, left - share, high
        steps.append(f"basamak {i + 1} , {share} m3 , birim {u} , ara toplam {total}")
        if left <= 0:
            break
    if r() < slip:
        steps.append(f"sabit bedel atlandi ara toplam {total}")
    else:
        total += FIXED_FEE
        steps.append(f"sabit bedel {FIXED_FEE} ara toplam {total}")
    discount = 0 if r() < slip else SUBSCRIBER_TYPES[sub_type]
    total = (total * (100 - discount)) // 100
    return " . ".join(steps + [f"indirim {discount} sonuc {total}"])


def _one_move(m3, sub_type, r):                 # shortcut patterns that skip the tiers
    d, disc = next(u for h, u in TIERS if m3 <= h), SUBSCRIBER_TYPES[sub_type]
    options = [exact_answer(m3, sub_type), ((m3 * d + FIXED_FEE) * (100 - disc)) // 100,
               tiered_total(m3) + FIXED_FEE, (tiered_total(m3) * (100 - disc)) // 100,
               m3 * TIERS[0][1] + FIXED_FEE]
    return options[pick(r(), [0.16, 0.30, 0.22, 0.20, 0.12])]


def call(prompt, seed=SEED, temperature=0.6, max_out=MAX_OUT):
    COUNTER[0] += 1
    toks = tokenize(prompt)
    if len(toks) > WINDOW:
        return {"status": "context_overflow", "input": len(toks), "output": 0, "response": ""}
    if COUNTER[0] % 37 == 0:
        return {"status": "rate_limit", "input": len(toks), "output": 0, "response": ""}
    r = rng(seed ^ trace_id(prompt))
    stepwise = "adim" in toks                   # does the prompt ask for a step-by-step solution
    if "hesap" in toks:                         # multi-step calculation task
        m3, sub_type = _parse_task(toks)
        y = (_step_by_step(m3, sub_type, r) if stepwise
             else f"sonuc {_one_move(m3, sub_type, r)}")
    else:                                       # single-step classification task
        p = {k: 0.15 for k in TOPICS}           # the label set is already written in the prompt
        for t in toks:
            for table, w in ((CUES, 1.0), (TRAPS, 1.3)):
                if table.get(t) in p:
                    p[table[t]] += w
        e = [math.exp(p[a] / temperature) for a in sorted(p)]
        k = sorted(p)[pick(r(), [v / sum(e) for v in e])]
        y = RATIONALE.format(topic=k) if stepwise else k
    out = tokenize(y)
    if len(out) > max_out:                      # stop condition
        out = out[:max_out]
        y = " ".join(out)
    return {"status": "ok", "input": len(toks), "output": len(out), "response": y}


def read_label(y):                              # the first label in the response
    return next((s for s in y.split() if s in TOPICS), None)


def read_number(y):                             # the last number in the response
    return next((int(x) for x in reversed(y.split()) if x.isdigit()), None)


TARIFF = ("Tarife: ilk 10 m3 birim 3 , sonraki 15 m3 birim 5 , sonraki 15 m3 birim 8 , "
          "40 ustu birim 12 . Sabit bedel 20 . Abone tipi indirimi yuzde konut 0 , "
          "tarim 20 , resmi 10 . Tutari hesapla . ")
CALC_BEFORE = TARIFF + "Son satira sadece sonucu yaz . Talep: "
CALC_STEP = TARIFF + "Once her basamagi ayri ayri adim adim yaz , sonra sonucu yaz . Talep: "
CLASS_BEFORE = ("Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife , kesinti . "
                "Yanit olarak tek etiket yaz . Talep: ")
CLASS_STEP = ("Talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife , kesinti . "
              "Once adim adim incele , sonra yalniz etiketi yaz . Talep: ")
x = CASES[0]
print(f"FICTIONAL calculation set {len(CASES)} records, FICTIONAL ticket set {len(TICKETS)} records")
print(f"example {x['id']}: {x['text']} , correct answer {exact_answer(x['m3'], x['type'])}")
for name, prompt, u in (("previous", CALC_BEFORE, MAX_OUT), ("stepwise", CALC_STEP, MAX_OUT_STEPS)):
    s = call(prompt + x["text"], max_out=u)
    s2 = call(prompt + x["text"], seed=SEED + 1, max_out=u)
    print(f"\n{name} prompt, input {s['input']} output {s['output']} tokens: {s['response']}")
    print(f"  same prompt seed {SEED + 1}: {read_number(s2['response'])}")
t = TICKETS[2]
print(f"\nclassification example {t['id']} , correct label {t['topic']}:")
for name, prompt, u in (("previous", CLASS_BEFORE, MAX_OUT), ("stepwise", CLASS_STEP, MAX_OUT_STEPS)):
    s = call(prompt + t["text"], max_out=u)
    print(f"  {name} prompt , {s['input']}+{s['output']} tokens: {s['response']}")

FICTIONAL calculation set 60 records, FICTIONAL ticket set 120 records
example 7001: tuketim 14 m3 abone tipi resmi tutar ne kadar , correct answer 63

previous prompt, input 71 output 2 tokens: sonuc 62
  same prompt seed 20260219: 70

stepwise prompt, input 77 output 43 tokens: basamak 1 , 10 m3 , birim 3 , ara toplam 30 . basamak 2 , 4 m3 , birim 5 , ara toplam 50 . sabit bedel 20 ara toplam 70 . indirim 10 sonuc 63
  same prompt seed 20260219: 63

classification example 4003 , correct label ariza:
  previous prompt , 50+1 tokens: ariza
  stepwise prompt , 55+33 tokens: birinci adim : talep metnindeki isaretler tarandi . ikinci adim : en guclu isaret ariza kavramini gosteriyor . ucuncu adim : etiket ariza

The one-move response said 62, the correct answer is 63; on the second seed it said 70. Both are wrong and there is no intermediate step in between. The stepwise response wrote the tiers one by one: the first 10 m³ at unit 3 gave 30, the remaining 4 m³ at unit 5 gave 20, with the fixed fee 70, and after the discount is deducted 63 — the same on both seeds. The cost is on the same line: the output tokens went from 2 to 43.

The last two lines are the same change on the other task. In classification, asking for step by step did not change the label, but the output tokens went from 1 to 33. The thirty-three-token rationale carries no information that the one-token response does not.

The Same Change on Two Tasks

The measurement is repeated on the whole set. Each prompt is run on three seeds and the number of calls is equal on both paths.

def measure(dataset, prompt, seed, max_out, correct, read_response):
    COUNTER[0] = 0
    n_in = n_out = hits = 0
    for x in dataset:
        s = call(prompt + x["text"], seed=seed, max_out=max_out)
        n_in, n_out, hits = n_in + s["input"], n_out + s["output"], hits + (read_response(s["response"]) == correct(x))
    return {"accuracy": hits / len(dataset), "input": n_in, "output": n_out, "total": n_in + n_out}


def report(title, dataset, prompt1, prompt2, correct, read_response):
    A = [measure(dataset, prompt1, SEED + j, MAX_OUT, correct, read_response) for j in range(3)]
    B = [measure(dataset, prompt2, SEED + j, MAX_OUT_STEPS, correct, read_response) for j in range(3)]
    print(f"{title:<26} {'previous':>9} {'stepwise':>9} {'diff':>9}")
    for j in range(3):
        print(f"{'accuracy seed ' + str(SEED + j):<26} {A[j]['accuracy']:>9.4f} "
              f"{B[j]['accuracy']:>9.4f} {B[j]['accuracy'] - A[j]['accuracy']:>+9.4f}")
    band = [max(x['accuracy'] for x in v) - min(x['accuracy'] for x in v) for v in (A, B)]
    print(f"{'three-run band':<26} {band[0]:>9.4f} {band[1]:>9.4f}")
    for name, k in (("input tokens", "input"), ("output tokens", "output"),
                    ("total tokens", "total")):
        print(f"{name:<26} {A[0][k]:>9d} {B[0][k]:>9d} {B[0][k] - A[0][k]:>+9d}")
    print(f"tokens {B[0]['total'] / A[0]['total']:.2f}x, per correct response "
          f"{A[0]['total'] / A[0]['accuracy'] / len(dataset):.1f} -> "
          f"{B[0]['total'] / B[0]['accuracy'] / len(dataset):.1f} tokens\n")


report("multi-step calculation", CASES, CALC_BEFORE, CALC_STEP, lambda x: exact_answer(x["m3"], x["type"]), read_number)
report("single-step classification", TICKETS, CLASS_BEFORE, CLASS_STEP, lambda x: x["topic"], read_label)

multi-step calculation      previous  stepwise      diff
accuracy seed 20260218        0.3000    0.8833   +0.5833
accuracy seed 20260219        0.3500    0.8833   +0.5333
accuracy seed 20260220        0.3000    0.9000   +0.6000
three-run band                0.0500    0.0167
input tokens                    4260      4620      +360
output tokens                    118      3334     +3216
total tokens                    4378      7954     +3576
tokens 1.82x, per correct response 243.2 -> 150.1 tokens

single-step classification  previous  stepwise      diff
accuracy seed 20260218        0.7750    0.7250   -0.0500
accuracy seed 20260219        0.7000    0.6583   -0.0417
accuracy seed 20260220        0.7333    0.8083   +0.0750
three-run band                0.0750    0.1500
input tokens                    5300      5900      +600
output tokens                    192      4023     +3831
total tokens                    5492      9923     +4431
tokens 1.81x, per correct response 59.1 -> 114.1 tokens

The upper table is where the technique works. Accuracy rises from 0.3000 to 0.8833 and the difference is large with the same sign on all three runs: +0.5833, +0.5333, +0.6000. The run bands are 0.0500 and 0.0167; the difference is more than ten times the band, so it counts as measured. Total tokens grow 1.82 times, almost all of the increase coming from the output; on the other hand the cost per correct response falls from 243.2 to 150.1 tokens.

The lower table is the task where the same technique does not pay. The difference is -0.0500, -0.0417 and +0.0750; not even the sign is stable, and all three stay inside the bands of 0.0750 and 0.1500. By the rule of the course this difference is unmeasured: the wobble between runs of the same prompt is larger than the wobble the change adds. On the cost side there is no uncertainty: total tokens grow 1.81 times and the cost per correct response rises from 59.1 to 114.1 tokens.

Where the Gain Comes From

What creates the difference is the structure of the task. The tariff calculation is chained; writing the intermediate steps breaks the chain into pieces and the error of each piece stays small. Classification is single-step: the cues are read and the label is chosen. There is no intermediate step to write; the rationale text does not produce the decision, it narrates it. The cost, on the other hand, is paid the same on both tasks, because a token does not know whether the intermediate step helped.

The rule of the course can be read from here: a prompt changed without measuring is not an improvement but a guess.

Summary

• Chain of thought means asking for the intermediate steps that lead to the result: a one-sentence prompt change, with the model and the set held fixed.
• On the multi-step fictional tariff calculation accuracy rises from 0.3000 to 0.8833; the difference is large with the same sign on all three runs and well outside the run band (0.0500 / 0.0167).
• On single-step classification the differences are -0.0500, -0.0417 and +0.0750; the sign changes and they stay inside the run band (0.0750 / 0.1500). This difference counts as unmeasured.
• The cost is certain on both tasks: total tokens grow 1.82 and 1.81 times. The cost per correct response falls from 243.2 to 150.1 in calculation and rises from 59.1 to 114.1 in classification — the same technique makes one task cheaper and the other more expensive.

Next Step

Step-by-step solving paid off in the calculation, but in a single run it produced a single chain. The chain itself is uncertain too: the accuracy of the stepwise path wobbles between 0.8833 and 0.9000, and with a single chain in hand there is no way to know which one is right. The next lesson turns this into scaffolding: the same prompt is run many times and the responses are voted on. What will be measured is how the gain changes with the number of samples, how many times the number of calls grows, and the number of cases where the consensus is reached on a wrong answer.