Step-Back Prompting

The previous three lessons always asked the model for the same thing: more steps. The steps were written out inside a single response, duplicated and voted on, branched and pruned — and in all three they looked in the same direction, toward the detail of the ticket. In the tree lesson the error came from exactly there: a misleading word spoiled the first step, and no step asked “what should I have been looking at in this ticket”.

Step-back prompting goes the other way. Before asking for the solution it raises the level of abstraction: which general rule do tickets of this kind get solved by. The prompt is still text, but this time in two stages, and the output of the first is written into the input of the second. The cost is known up front — the number of calls doubles. What will be measured is what is gained in return.

• RT28. The task and the fiction are the same as in the previous lesson: 48 fictional support tickets, a fault code with three components, thirty-two codes in total. What is called is a simulator; no request goes to a real model endpoint. Seed 20260218, second run 20260219.
• RT29. The fictional principle pool is four general rules. When a principle is written at the head of the prompt, the weight the simulator gives to each cue changes; this fictional scaffolding is the measurable counterpart of what a real instruction does — an instruction selects evidence.
• RT30. In this fiction the correct principle is gozlem (observation): the code is chosen by the symptom and the area the ticket reports, not by the route through which the case was opened. The other three principles are wrong.
• RT31. The two stages are two calls. The response of the first stage is written verbatim at the head of the second prompt; there is no correction, verification or fallback in between.
• RT32. The options are written into the prompt as component lists; writing the thirty-two codes one by one leaves no room for the principle.
• RT33. The cost is written in four units: number of calls, input tokens, output tokens and failed calls. The criterion is all three components being correct at once in the code task, and the degree being correct in the degree task.

Principle First, Then Solution

The previous prompt is a single one: instruction, options and ticket go together. The changed prompt is split in two. The first call asks only which general rule applies; it sees the ticket but does not see the code options. The second call takes the text of the returned rule at its head and asks for the real job. The first block builds the simulator, the principle pool and the two stages on one ticket.

# SIMULATOR. This code does not call a real model endpoint; it is a language model
# simulator written with the standard library. The support tickets and the fault codes are FICTIONAL too.
import math

SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW = 240                        # context window, tokens


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(text):                 # FICTIONAL tokenizer: a piece has at most five characters
    out = []
    for piece in text.lower().replace(",", " , ").split():
        while len(piece) > 5:
            out.append(piece[:5])
            piece = piece[5:]
        out.append(piece)
    return out


def trace_id(text):                 # depends on the prompt's content, independent of the run
    h = 2166136261
    for c in text:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


AREAS = ["dagitim", "sayac", "abonelik", "saha"]
SYMPTOMS = {"dagitim": ["kesinti", "basinc", "sizinti", "tikanma"],
            "sayac": ["endeks", "bozukluk", "degisim", "okuma"],
            "abonelik": ["tarife", "fatura", "odeme", "kayit"],
            "saha": ["randevu", "erisim", "malzeme", "gecikme"]}
DEGREES = ["birinci", "ikinci"]
AREA_WORD = {"dagitim": "sebeke", "sayac": "olcum", "abonelik": "sozlesme", "saha": "ekip"}
TRAP_WORD = {"dagitim": "hatti", "sayac": "cihaz", "abonelik": "hesap", "saha": "servis"}
SYMPTOM_WORD = {"kesinti": "akmiyor", "basinc": "zayif", "sizinti": "islak", "tikanma": "dolmus",
                "endeks": "rakam", "bozukluk": "calismiyor", "degisim": "yenisi",
                "okuma": "girilmemis", "tarife": "basamak", "fatura": "tutar",
                "odeme": "borcum", "kayit": "musteri", "randevu": "saatte", "erisim": "kapali",
                "malzeme": "parca", "gecikme": "bekliyor"}
DEGREE_WORD = {"birinci": "acil", "ikinci": "normal"}
AREA_CUE = {AREA_WORD[a][:5]: a for a in AREAS}                # area cue, weak
TRAP_CUE = {TRAP_WORD[a][:5]: a for a in AREAS}                # channel cue, heavier
SYMPTOM_CUE = {SYMPTOM_WORD[s][:5]: s for a in AREAS for s in SYMPTOMS[a]}
DEGREE_CUE = {DEGREE_WORD[d][:5]: d for d in DEGREES}
PRINCIPLE_CUE = {"sorun": "gozlem", "ilgil": "gozlem", "durum": "gozlem",
                 "diye": "gozlem", "taraf": "gozlem", "yaziy": "gozlem",
                 "kayit": "kanal", "uzeri": "kanal", "acilm": "sira",
                 "bildi": "sira", "gerek": "sure", "deger": "sure"}
PRINCIPLES = {
    "gozlem": "ilke gozlem : kod , talebin bildirdigi belirtiye ve yere gore secilir .",
    "kanal": "ilke kanal : kod , kaydin acildigi kanala gore secilir .",
    "sure": "ilke sure : kod , bildirimin aciliyet duzeyine gore secilir .",
    "sira": "ilke sira : kod , talebin geldigi siraya gore secilir ."}
RULE = {"gozle": {"A": 2.4, "T": 0.2, "S": 2.8},       # key: first piece of the principle name
        "kanal": {"A": 0.3, "T": 2.6}, "sure": {"S": 0.6}, "sira": {"A": 0.4, "S": 0.5}}

TICKETS = []
for i in range(48):
    r = rng(SEED + i)
    area = AREAS[int(r() * 4)]
    symptom = SYMPTOMS[area][int(r() * 4)]
    degree = DEGREES[int(r() * 2)]
    trap = AREAS[(AREAS.index(area) + 1 + int(r() * 3)) % 4]    # the channel in the ticket, misleading
    if r() < 0.5:
        text = (f"{TRAP_WORD[trap]} icin kayit acilmisti ama sorun {AREA_WORD[area]} tarafinda , "
                f"{SYMPTOM_WORD[symptom]} durumu var , {DEGREE_WORD[degree]} bakilmasi gerekiyor")
    else:
        text = (f"{TRAP_WORD[trap]} uzerinden bildirdim , {AREA_WORD[area]} ile ilgili "
                f"{SYMPTOM_WORD[symptom]} diye yaziyor , {DEGREE_WORD[degree]} olarak degerlendirin")
    TICKETS.append({"id": 5001 + i, "text": text, "code": (area, symptom, degree)})

ALL_CODES = [(a, s, d) for a in AREAS for s in SYMPTOMS[a] for d in DEGREES]
COUNTER = [0, 0, 0, 0]              # calls, input tokens, output tokens, failed calls


def join_path(p):
    return " ".join(p)


def call(prompt, options, seed=SEED, temperature=0.6):
    COUNTER[0] += 1
    toks = tokenize(prompt)
    COUNTER[1] += len(toks)
    if len(toks) > WINDOW or COUNTER[0] % 37 == 0:
        COUNTER[3] += 1
        return {"input": len(toks), "response": None}
    weight = {"A": 1.0, "T": 1.3, "S": 1.6, "D": 0.9}
    for t in toks:
        weight.update(RULE.get(t, {}))          # the principle in the prompt changes the cue weights
    r = rng(seed ^ trace_id(prompt))
    p = {k: 0.15 for k in options}
    for t in toks:
        for table, w in ((AREA_CUE, weight["A"]), (TRAP_CUE, weight["T"]),
                         (SYMPTOM_CUE, weight["S"]), (DEGREE_CUE, weight["D"]),
                         (PRINCIPLE_CUE, 1.0)):
            h = table.get(t)
            if h is not None:
                for k in p:
                    if h in k:
                        p[k] += w
    d = sorted(p, key=join_path)
    if temperature <= 0.0:
        k = max(d, key=lambda a: p[a])
    else:
        e = [math.exp(p[a] / temperature) for a in d]
        k = d[pick(r(), [v / sum(e) for v in e])]
    COUNTER[2] += len(tokenize(join_path(k)))
    return {"input": len(toks), "response": k}


CODE_PROMPT = ("Talebe bir ariza kodu ata . alanlar : " + " , ".join(AREAS)
               + " . belirtiler : " + " , ".join(s for a in AREAS for s in SYMPTOMS[a])
               + " . dereceler : " + " , ".join(DEGREES) + " . Talep : ")
PRINCIPLE_PROMPT = ("Cozmeden once bu talebin hangi genel kurala girdigini yaz . kurallar : "
                    + " , ".join(sorted(PRINCIPLES)) + " . Talep : ")

x = TICKETS[2]
print(f"FICTIONAL: {len(TICKETS)} tickets, {len(ALL_CODES)} fault codes, "
      f"{len(PRINCIPLES)} general rules")
print(f"ticket {x['id']}: {x['text']}")
print(f"correct code: {join_path(x['code'])}")
t1 = call(CODE_PROMPT + x["text"], ALL_CODES)
print(f"single stage: input {t1['input']} tokens, response '{join_path(t1['response'])}'")
i1 = call(PRINCIPLE_PROMPT + x["text"], [(k,) for k in sorted(PRINCIPLES)])
print(f"\nfirst stage: input {i1['input']} tokens, principle '{join_path(i1['response'])}'")
for name in ("gozlem", "kanal"):
    t2 = call(PRINCIPLES[name] + " " + CODE_PROMPT + x["text"], ALL_CODES)
    print(f"second stage principle {name}: input {t2['input']} tokens, "
          f"response '{join_path(t2['response'])}'")
u = call(PRINCIPLES["gozlem"] + " Talebe bir ariza kodu ata . secenekler : "
         + " , ".join(join_path(k) for k in ALL_CODES) + " . Talep : " + x["text"], ALL_CODES)
print(f"\nif the thirty-two codes were written one by one: input {u['input']} tokens "
      f"(limit {WINDOW}), response {u['response']}")

FICTIONAL: 48 tickets, 32 fault codes, 4 general rules
ticket 5003: hesap uzerinden bildirdim , olcum ile ilgili rakam diye yaziyor , acil olarak degerlendirin
correct code: sayac endeks birinci
single stage: input 100 tokens, response 'abonelik fatura birinci'

first stage: input 49 tokens, principle 'gozlem'
second stage principle gozlem: input 118 tokens, response 'sayac endeks ikinci'
second stage principle kanal: input 115 tokens, response 'abonelik odeme birinci'

if the thirty-two codes were written one by one: input 253 tokens (limit 240), response None

The single-stage prompt says abonelik fatura birinci: the word “hesap” (account) in the ticket is the route through which the case was opened and its cue is heavy; the real evidence, “olcum” (measurement), stayed beneath it. The first stage spends 49 tokens and finds the rule gozlem (observation); when the rule is written at the head of the second prompt the response becomes sayac endeks ikinci — area and symptom were fixed, the degree is still wrong. The rule says nothing about the degree, and the remaining error sits exactly there.

The third line is the other side of the cost. When a wrong principle — kanal (route) — is written at the head of the second prompt, the response becomes abonelik odeme birinci; worse than the single-stage one. The second stage does not make a correction of its own, it obeys the rule put in front of it. The last line is a design constraint: the previous lesson’s prompt wrote the thirty-two codes one by one, and when a principle is added at its head the input rises to 253 tokens and exceeds the 240 limit, and the call returns without producing any response. Step-back prompting needs room to have been opened in the window; this is why the options were written as a component list.

What the Two Stages Add and Take

The second block measures two tasks: the three-component code and the degree alone — the only component written word for word in the ticket. Each task is run single-stage and with a step back, each arrangement in two runs. At the end the source of the gain is separated: in how many tickets the first stage found the right principle, and what the second stage gives when each principle is imposed one by one.

DEGREE_PROMPT = "Talebin derecesini yaz . dereceler : " + " , ".join(DEGREES) + " . Talep : "
PRINCIPLE_OPTIONS = [(k,) for k in sorted(PRINCIPLES)]
CODE, DEGREE = lambda x: x["code"], lambda x: (x["code"][2],)


def single(prompt, options):
    return lambda x, t: (call(prompt + x["text"], options, t)["response"], None)


def step_back(prompt, options):                  # first the principle, then the solution
    def f(x, t):
        i = call(PRINCIPLE_PROMPT + x["text"], PRINCIPLE_OPTIONS, t)
        name = i["response"][0] if i["response"] else "sira"
        return call(PRINCIPLES[name] + " " + prompt + x["text"], options, t)["response"], name
    return f


def measure(f, target, seed=SEED):
    COUNTER[:] = [0, 0, 0, 0]
    hits = right_n = right_hits = wrong_hits = 0
    for x in TICKETS:
        y, name = f(x, seed)
        ok = y == target(x)
        hits += ok
        if name is not None:
            right_n += name == "gozlem"
            right_hits += ok and name == "gozlem"
            wrong_hits += ok and name != "gozlem"
    n = len(TICKETS)
    return {"accuracy": hits / n, "n_right": right_n,
            "right_acc": right_hits / right_n if right_n else 0.0,
            "wrong_acc": wrong_hits / (n - right_n) if right_n < n else 0.0,
            "calls": COUNTER[0], "input": COUNTER[1], "output": COUNTER[2],
            "failed": COUNTER[3]}


ARRANGEMENTS = [("code single stage", single(CODE_PROMPT, ALL_CODES), CODE),
                ("code step back", step_back(CODE_PROMPT, ALL_CODES), CODE),
                ("degree single stage", single(DEGREE_PROMPT, [(d,) for d in DEGREES]), DEGREE),
                ("degree step back", step_back(DEGREE_PROMPT, [(d,) for d in DEGREES]), DEGREE)]

print(f"{'prompt arrangement':<20}{'accuracy':>8}{'run2':>8}{'diff':>8}{'calls':>7}"
      f"{'input':>8}{'output':>7}{'failed':>10}")
results = {}
for name, f, target in ARRANGEMENTS:
    A, B = measure(f, target), measure(f, target, SEED + 1)
    results[name] = A
    print(f"{name:<20}{A['accuracy']:>8.4f}{B['accuracy']:>8.4f}"
          f"{B['accuracy'] - A['accuracy']:>+8.4f}{A['calls']:>7d}{A['input']:>8d}"
          f"{A['output']:>7d}{A['failed']:>10d}")
for task in ("code", "degree"):
    a, b = results[task + " single stage"], results[task + " step back"]
    ta, tb = a["input"] + a["output"], b["input"] + b["output"]
    print(f"\ncost of the {task} task: calls {a['calls']} -> {b['calls']}, "
          f"tokens {ta} -> {tb} ({tb / ta:.2f}x)")
K = results["code step back"]
print(f"principle stage: right principle {K['n_right']}/{len(TICKETS)}, code accuracy under the "
      f"right principle {K['right_acc']:.4f}, under a wrong one {K['wrong_acc']:.4f} "
      f"({len(TICKETS) - K['n_right']} tickets)")
print("code accuracy when a principle is imposed:", end="")
for name in sorted(PRINCIPLES):
    A = measure(lambda x, t, a=name: (call(PRINCIPLES[a] + " " + CODE_PROMPT + x["text"],
                                           ALL_CODES, t)["response"], None), CODE)
    print(f" {name} {A['accuracy']:.4f}", end="")
print()

prompt arrangement  accuracy    run2    diff  calls   input output    failed
code single stage     0.5417  0.4792 -0.0625     48    4889    249         1
code step back        0.7500  0.6458 -0.1042     96    8185    339         2
degree single stage   0.7708  0.7500 -0.0208     48    1913     94         1
degree step back      0.7708  0.7917 +0.0208     96    5209    186         2

cost of the code task: calls 48 -> 96, tokens 5138 -> 8524 (1.66x)

cost of the degree task: calls 48 -> 96, tokens 2007 -> 5395 (2.69x)
principle stage: right principle 45/48, code accuracy under the right principle 0.8000, under a wrong one 0.0000 (3 tickets)
code accuracy when a principle is imposed: gozlem 0.7708 kanal 0.0000 sira 0.0625 sure 0.1458

In the code task the two-stage prompt goes from 0.5417 to 0.7500, and in the second run from 0.4792 to 0.6458. The gain is +0.2083 and +0.1667 in the two runs; the arrangements’ own two-run differences are 0.0625 and 0.1042. Since the gain is larger than both, this difference counts as measured — not all the techniques in this topic pass this test. The cost is certain too: calls from 48 to 96, input tokens from 4889 to 8185, total tokens 1.66 times, the cost per correct code from 198 to 237 tokens. The technique both worked and became more expensive.

The last two lines tell the source of the gain and the real risk of the technique. The first stage found the right rule in 45 of 48 tickets, and code accuracy in those 45 is 0.8000. In all three tickets where a wrong rule was chosen, the code came out wrong — a small number, but the last line proves it alone. When the four rules are imposed in turn, the second stage gives 0.7708 with gozlem, 0.1458 with sure, 0.0625 with sira and 0.0000 with kanal; three of them are below the single-stage 0.5417. Step-back prompting does not add reasoning to the model, it adds a new place that can be wrong to the prompt.

The bottom two rows are the limit of the technique. In the degree task the single-stage prompt gives 0.7708 and 0.7500, the step-back prompt 0.7708 and 0.7917; the difference is 0.0000 in one run and +0.0417 in the other, and both are of the order of the arrangements’ own run differences. Counts as unmeasured. In return the calls double again and the tokens rise 2.69 times — a worse ratio than in the code task, because as the actual prompt gets shorter the share of the principle text grows. The degree is written word for word in the ticket; the solution does not get stuck at the wrong level of abstraction. Step-back prompting pays off when the source of the error is the choice of evidence; if the evidence is already chosen correctly, it is only twice the calls.

Summary

• Step-back prompting splits the call in two: the first call asks for the general rule the ticket falls under, the second call writes that rule at its head and asks for the solution. There is no correction in between.
• In the code task accuracy goes from 0.5417 to 0.7500 (in the second run from 0.4792 to 0.6458), and since the gain is larger than the arrangements’ two-run differences it counts as measured.
• The cost is certain: calls from 48 to 96, total tokens 1.66 times, the cost per correct code from 198 to 237 tokens.
• The gain depends on the first stage: the right rule was found in 45 tickets and accuracy there is 0.8000. When the rules are imposed one by one the second stage varies between 0.7708 and 0.0000; all three wrong rules are below the single-stage prompt.
• In the task that asks only for the degree the difference stays between 0.0000 and +0.0417 and cannot be measured, while the tokens rise to 2.69 times. Raising the level of abstraction has no return for a component that is written word for word.

Next Step

Four lessons built four separate prompt arrangements, and all four tried to win from the same place: the model taking more steps inside itself. In all of them the steps were taken entirely inside the simulator; the only thing coming from outside was the text of the ticket, and that text was already at hand in the first call.

In this lesson the limit of that appeared. What the right rule was was not written in the ticket; the first stage derived it from the simulator’s own tendency and was wrong on three tickets. A situation where the rule could be read from outside — looking into a fictional document pool, querying a record — was never covered. A prompt arrangement in which one of the steps is looking outside is the topic of the next lesson: asking for thought and action in the same prompt, and in how many responses this produces a valid action.