Error Recovery

One column stayed silent from start to finish in the cost table of the previous lesson: the tool error. Errors occurred in all six setups, and in none of them was it written what to do. When the loop of the shared definition sees an error, it turns toward the same target again; this is a behavior but not a policy, because it was not chosen. The retry, backoff, and circuit breaker procedure was measured in the Resilience and Reliability course; it is not repeated. The only thing measured is what the failed tool does to the loop.

• AQ51. The environment, tools, tasks, and loop are fictional. The production stream is the same as that of the previous lesson’s: twenty-four tasks run ten times, the budget is 6, and the total is 240 runs. The seeds are 20260218 and 20260219.
• AQ52. Stop: the loop ends at the first error; this is the baseline. Retry: the target does not change; at the next step the loop turns to the same target again. Skip: the tool that returned an error becomes unselectable for the rest of the run; if the draw gives it, the next tool in the list is used.
• AQ53. There is no second tool in the set that does the same job; “moving on to another tool” means dropping the failed tool, and if the plan needs it, the plan cannot be completed.

Three Policies, One Stream

The block sets up the shared definition: the fictional environment, five tools, twenty-four tasks, and the loop.

# FICTIONAL. The measurement network's subscriber records, document set, five tools and
# twenty-four tasks are defined within the lesson; no real endpoint or tool server is called.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in str(m).lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def trace_id(m):
    h = 2166136261
    for c in str(m):
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
SUBSCRIBERS = {}
for i in range(60):
    r = rng(SEED + 31 * i)
    previous = 10000 + int(r() * 80000)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "previous": previous, "current": previous + int(r() * 70)}
TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16
COMMERCIAL_FACTOR = 2
DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu",
             "TR02": "isyeri abonesinde birim ucret iki katidir",
             "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir",
             "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider",
             "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"}

CALLS = [0]                              # tool call counter


def _charge(m3, type_):
    remaining, lower, total = m3, 0, 0
    for upper, unit in TIERS:
        band = min(remaining, upper - lower)
        if band <= 0:
            break
        total += band * unit
        remaining -= band
        lower = upper
    total += remaining * ABOVE
    return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1)


def t_read_meter(subscriber=None, **_):
    a = SUBSCRIBERS.get(subscriber)
    if a is None:
        return {"error": "no such subscriber"}
    return {"previous": a["previous"], "current": a["current"], "type": a["type"],
            "region": a["region"]}


def t_consumption(previous=None, current=None, **_):
    if previous is None or current is None:
        return {"error": "missing field"}
    f = current - previous
    return {"m3": f + 100000 if f < 0 else f}


def t_tariff(m3=None, type="residential", **_):
    if m3 is None:
        return {"error": "missing field"}
    return {"amount": _charge(m3, type)}


def t_document(query=None, **_):
    if not query:
        return {"error": "missing field"}
    q = set(tokenize(query))
    best, code = -1, None
    for k, m in DOCUMENTS.items():
        o = len(q & set(tokenize(m)))
        if o > best:
            best, code = o, k
    return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"}


def t_record(subscriber=None, column=None, value=None, **_):
    if not (subscriber and column and value is not None):
        return {"error": "missing field"}
    return {"written": True, "subscriber": subscriber, "column": column, "value": value}


TOOLS = {
    "read_meter":    {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False},
    "consumption":   {"function": t_consumption, "fields": ("previous", "current"),
                      "side_effect": False},
    "tariff":        {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False},
    "search_docs":   {"function": t_document, "fields": ("query",), "side_effect": False},
    "write_record":  {"function": t_record, "fields": ("subscriber", "column", "value"),
                      "side_effect": True},
}
TOOL_NAMES = list(TOOLS)


def call_tool(name, state):
    """Fields outside the schema are filtered out; a missing field is the tool's own error."""
    CALLS[0] += 1
    if name not in TOOLS:
        return {"error": "no such tool"}
    kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state}
    return TOOLS[name]["function"](**kw)


TASKS = []
for i in range(24):
    r = rng(SEED + 9973 * i)
    sub = f"A{1 + int(r() * 60):03d}"
    t = pick(r(), [0.30, 0.22, 0.20, 0.28])
    if t == 0:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub,
                      "request": f"{sub} abonesinin bu donem bedeli nedir",
                      "sequence": ("read_meter", "consumption", "tariff")})
    elif t == 1:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub,
                      "request": f"{sub} abonesinin bu donem tuketimi kac metrekup",
                      "sequence": ("read_meter", "consumption")})
    elif t == 2:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub,
                      "request": "sayac donusunde endeks farki nasil hesaplanir",
                      "sequence": ("search_docs",)})
    else:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub,
                      "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz",
                      "sequence": ("read_meter", "consumption", "tariff", "write_record")})


def correct_answer(g):
    a = SUBSCRIBERS[g["subscriber"]]
    f = a["current"] - a["previous"]
    m3 = f + 100000 if f < 0 else f
    if g["kind"] == "consumption":
        return m3
    if g["kind"] == "document":
        return "SY01"
    return _charge(m3, a["type"])


def selection(d):
    return max(0.34, 0.80 - 0.09 * (d - 1))       # share of picking the right tool, falls with depth


def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False):
    """FICTIONAL agent loop. No real model is called; tool selection
    comes from a single draw that depends on the task's sequence and the depth."""
    r = rng(seed ^ trace_id(g["name"]))
    state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"}
    trace, step, tok, errors = [], 0, 0, 0
    seq, k = g["sequence"], 0
    while step < max_steps:
        step += 1
        tok += 60 + 22 * len(trace)
        target = seq[k] if k < len(seq) else None
        if target is None:                         # the job is done , the agent should stop
            if r() < 0.88:
                return {"done": True, "steps": step, "trace": trace, "tokens": tok,
                        "errors": errors, "answer": state.get("result"),
                        "approvals": state.get("approvals", 0)}
            choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))]
        else:
            choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))]
        if TOOLS[choice]["side_effect"] and approval:
            state["approvals"] = state.get("approvals", 0) + 1
        if choice == "write_record":
            state["value"] = state.get("result")
        s = call_tool(choice, state)
        trace.append((choice, "error" in s))
        tok += 30
        if "error" in s:
            errors += 1
            if self_critique:
                tok += 40
            continue
        state.update({kk: vv for kk, vv in s.items() if kk != "error"})
        if choice == "consumption":
            state["result"] = s["m3"]
        if choice == "tariff":
            state["result"] = s["amount"]
        if choice == "search_docs":
            state["result"] = s["code"]
        if choice == target:
            k += 1
    return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors,
            "answer": state.get("result"), "approvals": state.get("approvals", 0)}

The policy changes only the error branch.

# The recovery policy changes only the error branch of the shared loop.
BUDGET, RUNS = 6, 10
RESULT_FIELD = {"consumption": "m3", "tariff": "amount", "search_docs": "code"}   # tools that write the result


def agent_policy(g, policy, max_steps=BUDGET, seed=SEED):
    r = rng(seed ^ trace_id(g["name"]))
    state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"}
    trace, step, tok, errors, banned = [], 0, 0, 0, set()
    seq, k = g["sequence"], 0
    while step < max_steps:
        step += 1
        tok += 60 + 22 * len(trace)
        target = seq[k] if k < len(seq) else None
        if target is None:
            if r() < 0.88:
                break
            choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))]
        else:
            choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))]
        while choice in banned and len(banned) < len(TOOL_NAMES):
            choice = TOOL_NAMES[(TOOL_NAMES.index(choice) + 1) % len(TOOL_NAMES)]
        if choice == "write_record":
            state["value"] = state.get("result")
        s = call_tool(choice, state)
        trace.append((choice, "error" in s))
        tok += 30
        if "error" in s:
            errors += 1
            if policy == "stop":
                break
            if policy == "skip":
                banned.add(choice)
            continue
        state.update({kk: vv for kk, vv in s.items() if kk != "error"})
        if choice in RESULT_FIELD:
            state["result"] = s[RESULT_FIELD[choice]]
        if choice == target:
            k += 1
    return {"steps": step, "tokens": tok, "errors": errors, "answer": state.get("result")}


FIELDS = ("steps", "tokens", "errors", "answer")
print("retry is the same as the shared agent:",
      all(agent_policy(g, "retry") == {a: agent(g, BUDGET, SEED)[a] for a in FIELDS} for g in TASKS))
print("\nseed      policy    success steps errors    tokens steps per correct answer")
for base_seed in (20260218, 20260219):
    for p in ("stop", "retry", "skip"):
        correct, steps, tok, errors = 0, 0, 0, 0
        for d in range(RUNS):
            for g in TASKS:
                s = agent_policy(g, p, BUDGET, base_seed + 7 * d)
                steps += s["steps"]
                tok += s["tokens"]
                errors += s["errors"]
                correct += s["answer"] == correct_answer(g)
        n = RUNS * len(TASKS)
        print(f"{base_seed}  {p:<8} {correct / n:>8.4f} {steps:>5} {errors:>6} {tok:>9}"
              f" {steps / correct:>24.2f}")

retry is the same as the shared agent: True

seed      policy    success steps errors    tokens steps per correct answer
20260218  stop       0.7500   727     53     81996                     4.04
20260218  retry      0.9375   900     61    106352                     4.00
20260218  skip       0.8917   907     64    108030                     4.24
20260219  stop       0.7667   768     54     89180                     4.17
20260219  retry      0.9542   944     66    114430                     4.12
20260219  skip       0.8625   954     73    116776                     4.61

What the Policy Adds and What It Costs

The first line of the output is a check: the loop we used throughout the course had an unwritten policy, and it was retry.

Stop is the cheapest in all the cost columns: 727 steps, 53 tool errors, 81996 tokens. Its task success is 0.1875 below the 0.9375 of retry; since the difference between the two runs is 0.0167, the drop is ten times the band. Read per correct answer, the saving disappears: 4.04 steps for stop, 4.00 for retry, and the difference is within stop’s own difference between runs (4.04 against 4.17). Stopping lowers the spending of steps; it does not change the efficiency.

Skip is the example for the course’s second claim: intuition expects the errors to decrease, and the measurement gives the opposite. The tool errors rise from 61 to 64, and in the second run from 66 to 73; the steps go up to 907, and the task success falls to 0.8917. The mechanism works in two directions: the tool that gets banned has often been selected wrongly already, and the one that replaces it is wrong too, that is, the error slides onto another name; if a tool in the plan is banned, the plan ends. The loss is 0.0458 and 0.0917 in the two runs; since the widest band is 0.0292, both are above it, but the first is only one and a half times the band.

Summary

• The recovery policy is the error branch of the loop; even if it is not written, a policy runs, and here the one that runs is retry.
• Stop is the cheapest (727 steps, 53 errors, 81996 tokens), lowers task success by 0.1875, and does not improve step efficiency.
• Dropping the failed tool raises the tool errors to 64 and lowers task success to 0.8917.

Course Wrap-Up

The rule of the course is written line by line in the table: the number of an agent is the number of the whole loop, not of a single answer, and an agent whose stopping criterion is not written counts as unmeasured. The numbers in the without-scaffolding column are not low because of a bad model; they come out low because where the loop will stop, how the tool introduces itself, what will be remembered, and what happens when a step breaks are not written.

The second claim is read from the same table too: an added step can make the work worse. The rows that say it by name are these: Executing Tool Calls (truncation lowers 0.9583 to 0.2917 and raises the token to 15945), Web Search and Browsing (the secondary class lowers 0.7500 to 0.3750), Short-Term Memory (narrowing raises the token to 12026), Long-Term Memory (result only lowers 0.9167 to 0.5833), Context Compression (the tightest summarization spends 1.53 times the baseline), Multi-Agent Patterns (the second agent brings no gain), Keeping the Human in the Loop (spreading the scope raises the question to 74), Cost and Latency Control (the small class is expensive per answer), and this lesson (skip raises the errors to 64). In nine places the added step cost, because it was added.

Differences below the measurement band are also written by name and do not count as findings: in What Is an Agent the baseline’s single hit, in The Agent Loop the 0.0917 and 0.0160 of the neighboring depths, in The Planner and Executor Split the 0.0833 of the split, in Stopping Criteria the 0.0417 drop from six to eight, in Self-Critique and Verification the round differences, in Short-Term Memory the 0.0833 at the window of 200, in Memory Decay the gain of tightening the threshold below eight, and in this lesson the step efficiency of stop. None of them is zero, none of them is measured.

In twenty-two lessons the agent was built, equipped, and measured. One thing was never asked: every tool was defined by hand within a lesson. The name, description, and schema of read_meter were written in this course and only the agent here saw them. How the same tool would be handed to another agent, another team, or another application, who would hold the definition, and who would break when the interface changed were not asked. The next course turns the tool definition from a dictionary inside a lesson into a shared surface and counts its cost.