Self-Critique and Verification

In the previous four lessons the loop was built and where it stops was written. The agent picks a tool, writes the returned observation to its state, takes the next step by looking at it; it stops when its job is done or the budget is full. When it closes, an answer is left in hand.

The one thing that was not asked in four lessons is this: is that answer correct. When the agent stops, it does not turn back and look at its own trace. This lesson adds that missing round and measures two things: what the round catches and what the round costs. Running the same prompt several times and voting on the answers is a separate arrangement and was measured in the Prompt Engineering course; here there are no several answers to vote on, and the trace left by a single loop is looked at.

• AA41. The setup is common: the fictional subscriber records of the measurement network, the fictional document vocabulary, five fictional tools, twenty-four fictional tasks and an agent simulator written inside the lesson. No real model endpoint or tool server is called.
• AA42. The baseline is the scaffolding-free loop: the same set, the same budget, the round closed. The plain row with the budget doubled also stays in the table, because the steps the round spends could also have been given to a single longer run.
• AA43. The round cannot see the correct answer; it looks only at three observable signs: did the loop stop by itself, did a tool error occur, was an answer produced.
• AA44. If one of the three is broken, the task counts as flagged and the loop runs a second time. The round is single, and it can corrupt the answer as well as correct it.
• AA45. The cost of the round after every run is one step and 45 + 18 x len(trace) tokens; in a run with the round, every tool error adds 40 tokens for reading the trace.
• AA46. The second run seed is 20260219; in this set the run-to-run difference is one task (0.0417) and every difference below it counts as unmeasured.

The Three Signs the Round Can Look At

The round has no correct answer in hand; if it did, there would be no need for the agent. It has only the trace.

# COMMON FICTIONAL SETUP -- the 60 fictional subscribers of the measurement network,
# 5 fictional documents, 5 fictional tools, 24 fictional tasks and the agent simulator
# written inside the lesson. No real model endpoint, tool server or runtime is called.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in str(m).lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def trace_id(m):
    h = 2166136261
    for c in str(m):
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


# ---- FICTIONAL environment: records of the measurement network
REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
SUBSCRIBERS = {}
for i in range(60):
    r = rng(SEED + 31 * i)
    previous = 10000 + int(r() * 80000)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "previous": previous, "current": previous + int(r() * 70)}
TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16
COMMERCIAL_FACTOR = 2
DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu",
             "TR02": "isyeri abonesinde birim ucret iki katidir",
             "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir",
             "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider",
             "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"}

CALLS = [0]                              # tool call counter


def _charge(m3, type_):
    remaining, lower, total = m3, 0, 0
    for upper, unit in TIERS:
        band = min(remaining, upper - lower)
        if band <= 0:
            break
        total += band * unit
        remaining -= band
        lower = upper
    total += remaining * ABOVE
    return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1)


def t_read_meter(subscriber=None, **_):
    a = SUBSCRIBERS.get(subscriber)
    if a is None:
        return {"error": "no such subscriber"}
    return {"previous": a["previous"], "current": a["current"], "type": a["type"],
            "region": a["region"]}


def t_consumption(previous=None, current=None, **_):
    if previous is None or current is None:
        return {"error": "missing field"}
    f = current - previous
    return {"m3": f + 100000 if f < 0 else f}


def t_tariff(m3=None, type="residential", **_):
    if m3 is None:
        return {"error": "missing field"}
    return {"amount": _charge(m3, type)}


def t_document(query=None, **_):
    if not query:
        return {"error": "missing field"}
    q = set(tokenize(query))
    best, code = -1, None
    for k, m in DOCUMENTS.items():
        o = len(q & set(tokenize(m)))
        if o > best:
            best, code = o, k
    return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"}


def t_record(subscriber=None, column=None, value=None, **_):
    if not (subscriber and column and value is not None):
        return {"error": "missing field"}
    return {"written": True, "subscriber": subscriber, "column": column, "value": value}


TOOLS = {
    "read_meter":    {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False},
    "consumption":   {"function": t_consumption, "fields": ("previous", "current"),
                      "side_effect": False},
    "tariff":        {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False},
    "search_docs":   {"function": t_document, "fields": ("query",), "side_effect": False},
    "write_record":  {"function": t_record, "fields": ("subscriber", "column", "value"),
                      "side_effect": True},
}
TOOL_NAMES = list(TOOLS)


def call_tool(name, state):
    """Fields outside the schema are filtered out; a missing field is the tool's own error."""
    CALLS[0] += 1
    if name not in TOOLS:
        return {"error": "no such tool"}
    kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state}
    return TOOLS[name]["function"](**kw)


# ---- FICTIONAL tasks: each one is solved by a known tool sequence
TASKS = []
for i in range(24):
    r = rng(SEED + 9973 * i)
    sub = f"A{1 + int(r() * 60):03d}"
    t = pick(r(), [0.30, 0.22, 0.20, 0.28])
    if t == 0:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub,
                      "request": f"{sub} abonesinin bu donem bedeli nedir",
                      "sequence": ("read_meter", "consumption", "tariff")})
    elif t == 1:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub,
                      "request": f"{sub} abonesinin bu donem tuketimi kac metrekup",
                      "sequence": ("read_meter", "consumption")})
    elif t == 2:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub,
                      "request": "sayac donusunde endeks farki nasil hesaplanir",
                      "sequence": ("search_docs",)})
    else:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub,
                      "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz",
                      "sequence": ("read_meter", "consumption", "tariff", "write_record")})


def correct_answer(g):
    a = SUBSCRIBERS[g["subscriber"]]
    f = a["current"] - a["previous"]
    m3 = f + 100000 if f < 0 else f
    if g["kind"] == "consumption":
        return m3
    if g["kind"] == "document":
        return "SY01"
    return _charge(m3, a["type"])


# ---- FICTIONAL agent: at each step it picks a tool or finishes
def selection(d):
    return max(0.34, 0.80 - 0.09 * (d - 1))       # share of picking the right tool, falls with depth


def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False):
    """FICTIONAL agent loop. No real model is called; tool selection
    comes from a single draw that depends on the task's sequence and the depth."""
    r = rng(seed ^ trace_id(g["name"]))
    state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"}
    trace, step, tok, errors = [], 0, 0, 0
    seq, k = g["sequence"], 0
    while step < max_steps:
        step += 1
        tok += 60 + 22 * len(trace)
        target = seq[k] if k < len(seq) else None
        if target is None:                         # the job is done , the agent should stop
            if r() < 0.88:
                return {"done": True, "steps": step, "trace": trace, "tokens": tok,
                        "errors": errors, "answer": state.get("result"),
                        "approvals": state.get("approvals", 0)}
            choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))]
        else:
            choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))]
        if TOOLS[choice]["side_effect"] and approval:
            state["approvals"] = state.get("approvals", 0) + 1
        if choice == "write_record":
            state["value"] = state.get("result")
        s = call_tool(choice, state)
        trace.append((choice, "error" in s))
        tok += 30
        if "error" in s:
            errors += 1
            if self_critique:
                tok += 40
            continue
        state.update({kk: vv for kk, vv in s.items() if kk != "error"})
        if choice == "consumption":
            state["result"] = s["m3"]
        if choice == "tariff":
            state["result"] = s["amount"]
        if choice == "search_docs":
            state["result"] = s["code"]
        if choice == target:
            k += 1
    return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors,
            "answer": state.get("result"), "approvals": state.get("approvals", 0)}

# ---- Part added in this lesson: the self-critique round


def flagged(s):
    """The three signs the round can look at: did the loop stop by itself, did a
    tool error occur, was an answer produced. The round does not have the correct answer."""
    return (not s["done"]) or s["errors"] > 0 or s["answer"] is None


def review_cost(s):
    return 45 + 18 * len(s["trace"])


print("task   kind        done   error answer flagged  correct")
for g in TASKS[:6]:
    s = agent(g, max_steps=2)
    print(f"{g['name']:<7}{g['kind']:<12}{str(s['done']):<7}{s['errors']:<6}"
          f"{str(s['answer'] is not None):<7}{str(flagged(s)):<9}"
          f"{s['answer'] == correct_answer(g)}")


def four_cells(b, seed):
    c = {"fw": 0, "fr": 0, "sw": 0, "sr": 0}
    for g in TASKS:
        s = agent(g, max_steps=b, seed=seed)
        c[("f" if flagged(s) else "s")
          + ("r" if s["answer"] == correct_answer(g) else "w")] += 1
    return c


print("\nbudget      seed flagged+wrong  flagged+right   silent+wrong  silent+right")
for b in (2, 4, 8):
    for t in (SEED, SEED + 1):
        c = four_cells(b, t)
        print(f"{b:>6}{t:>10}{c['fw']:>14}{c['fr']:>15}{c['sw']:>15}{c['sr']:>14}")

print("\nsilent wrong answer (budget 8, seed 20260219):")
for g in TASKS:
    s = agent(g, max_steps=8, seed=SEED + 1)
    if not flagged(s) and s["answer"] != correct_answer(g):
        print(" ", g["name"], g["kind"], [a for a, h in s["trace"]])
        print("  answer", s["answer"], "-- correct", correct_answer(g))

task   kind        done   error answer flagged  correct
G01    document    False  2     False  True     False
G02    document    True   0     True   False    True
G03    consumption False  1     False  True     False
G04    consumption False  0     True   True     True
G05    charge      False  0     True   True     False
G06    charge      False  0     True   True     False

budget      seed flagged+wrong  flagged+right   silent+wrong  silent+right
     2  20260218            15              6              0             3
     2  20260219            14              7              0             3
     4  20260218             3              7              0            14
     4  20260219             4              6              0            14
     8  20260218             1              6              0            17
     8  20260219             0              6              1            17

silent wrong answer (budget 8, seed 20260219):
  G15 record ['read_meter', 'consumption', 'tariff', 'consumption', 'write_record']
  answer 57 -- correct 582

The first table shows why the round is not cheap. At a narrow budget five of the six tasks are flagged; four are really wrong, and one (G04) has produced a correct answer. A broken sign is not proof of a wrong answer, it is a reason to look.

The four-cell table confirms this on the whole set. At narrow budgets there is no silent wrong answer; the cost is unnecessary flags, seven against three wrong answers at four steps. At eight steps the table turns around, and the last lines of the block print the task that stayed silent. G15 is a record task; its sequence is complete, the loop stopped by itself, and no tool error occurred. But after the job was done the agent wandered and called the consumption tool again, this call overwrote the accumulated result with consumption instead of the charge, and that value was written to the record. The trace looks clean because every step is valid on its own; what is broken is the order of the steps.

What the Round Adds and Multiplies

The round is now run as an arrangement.

def run(b, seed, critique=False):
    """critique=False is the plain loop. critique=True: the loop ends, the round looks at
    the trace, and if flagged the same task runs once more in a second loop."""
    CALLS[0] = 0
    d = {"right": 0, "steps": 0, "tokens": 0, "flags": 0, "rescued": 0, "broke": 0}
    for g in TASKS:
        s = agent(g, max_steps=b, seed=seed, self_critique=critique)
        d["steps"] += s["steps"] + critique
        d["tokens"] += s["tokens"] + (review_cost(s) if critique else 0)
        first = s["answer"] == correct_answer(g)
        if critique and flagged(s):
            d["flags"] += 1
            s = agent(g, max_steps=b, seed=seed ^ trace_id("tur2|" + g["name"]),
                      self_critique=True)
            d["steps"] += s["steps"] + 1
            d["tokens"] += s["tokens"] + review_cost(s)
        final = s["answer"] == correct_answer(g)
        d["right"] += final
        d["rescued"] += final and not first
        d["broke"] += first and not final
    d["calls"] = CALLS[0]
    return d


for t in (SEED, SEED + 1):
    print(f"\nseed {t}")
    print("setup             success   step  calls    tokens   flags  rescued broke")
    for b in (2, 4, 6, 8):
        for label, critique in (("plain", False), ("round", True)):
            d = run(b, t, critique)
            extra = (f"{d['flags']:>8}{d['rescued']:>9}{d['broke']:>6}" if critique
                     else f"{'-':>8}{'-':>9}{'-':>6}")
            print(f"budget {b:>2}, {label}{d['right']:>6}/24{d['steps']:>7}"
                  f"{d['calls']:>7}{d['tokens']:>10}{extra}")

seed 20260218
setup             success   step  calls    tokens   flags  rescued broke
budget  2, plain     9/24     48     45      4758       -        -     -
budget  2, round    13/24    135     84     13007      21        4     0
budget  4, plain    21/24     84     64      9402       -        -     -
budget  4, round    23/24    152     94     17052      10        2     0
budget  6, plain    24/24     92     71     10884       -        -     -
budget  6, round    23/24    164    102     19671       9        0     1
budget  8, plain    23/24     97     74     11978       -        -     -
budget  8, round    23/24    155     94     18821       7        1     1

seed 20260219
setup             success   step  calls    tokens   flags  rescued broke
budget  2, plain    10/24     48     45      4758       -        -     -
budget  2, round    12/24    135     83     12839      21        4     2
budget  4, plain    20/24     83     67      9366       -        -     -
budget  4, round    22/24    151     96     16822      10        2     0
budget  6, plain    23/24     96     74     11610       -        -     -
budget  6, round    23/24    155     96     18493       7        0     0
budget  8, plain    23/24     98     74     11994       -        -     -
budget  8, round    23/24    152     92     18312       6        0     0

The budget of four steps is the row where the round looks its best: success goes from 21/24 to 23/24, two tasks on both runs. The cost: steps from 84 to 152, calls from 64 to 94, tokens from 9402 to 17052. Of the ten flags only two correct an answer.

The plain row at eight steps puts this gain in context: 23/24, 97 steps, 11978 tokens. The plain loop reaches in 97 steps where the round arrived with 152 steps; when the surplus is given to a longer single run, the same success is bought more cheaply. At the budget of two steps the gap is sharper: the run with the round gives 13/24 with 135 steps, while the plain run at four steps gives 21/24 with 84 steps.

The row at six steps is the clear example of the second claim of the course: the plain loop makes 24/24 there, and when the round is added it goes down to 23/24. None of the nine flags corrects a wrong answer, and one corrupts a correct one.

Below the Band

Not every difference in this table is a finding. The success of the plain loop is 24/24 and 23/24 on the two seeds at six steps and 23/24 and 23/24 at eight; the measurement band is one task. The one-task fall the round costs at six steps and the zero difference at eight are inside this band and count as unmeasured: the mechanism can be shown, its effect cannot be measured at this set size. The only gain that stays above the band is the budget of four steps. At eight steps the only concrete number of the round is the cost: seven flags, six of them unnecessary, and zero tasks gained.

Summary

• A stopped agent can look at its own trace but cannot see the correct answer; the three signs it can look at are whether it stopped by itself, whether a tool error occurred and whether it produced an answer.
• A broken sign is not proof of a wrong answer: at the budget of four steps only three of the ten flags correspond to a wrong answer.
• At wide budgets a wrong answer with a clean trace appears; a loop that stopped by itself without errors can overwrite the result it accumulated while wandering, and the round cannot see this.
• The highest gain of the round is two tasks at the budget of four steps (21/24 to 23/24); steps go from 84 to 152 and tokens from 9402 to 17052. The plain loop at eight steps gives the same 23/24 with 97 steps.
• The differences at six and eight steps are inside the band of two runs (one task) and are not counted as findings.

Next Step

In this lesson the one that checks and the one that is checked were the same loop, and the round stayed limited to the surface of the trace. One way to take the looking eye outside the loop is to split the job and give it to a second agent. The next lesson puts a single agent and two agents side by side on the same task set. What will be measured is not only success; the tokens of the handed over context and the number of fields lost in the handoff are counted separately, because the cost of a second agent is paid not in steps but in communication.