--- title: 'Error Recovery' source: 'https://academia.sh/en/courses/ai-agents/error-recovery' course: 'AI Agents' language: en updated: '2026-10-21T05:02:26+00:00' license: 'CC BY-SA 4.0' objectives: - 'Defines the three recovery policies on the error branch of the loop and counts task success, steps, and tool errors in each.' - 'Shows with its mechanism that disabling the failed tool raises the tool errors instead of lowering them.' - "Collects the without-scaffolding and with-scaffolding numbers of the course's twenty-two lessons in a single table." --- # Error Recovery Measuring what a failed tool call does to the loop with three recovery policies: in a production stream of two hundred forty runs, stopping at the first error gives a task success of 0.7500, 727 steps, 53 tool errors, and 81996 tokens; retrying the same tool raises success to 0.9375 and moves the steps to 900, the tool errors to 61, and the tokens to 106352. Dropping the failed tool and moving on to another does the opposite of what is expected: tool errors rise to 64, steps to 907, and success falls to 0.8917. The step efficiency that stopping gains stays within the band of the two runs. The lesson is the close of the AI Agents course and collects the without-scaffolding and with-scaffolding numbers of twenty-two lessons in a single table. One column stayed silent from start to finish in the cost table of the previous lesson: the tool error. Errors occurred in all six setups, and in none of them was it written what to do. When the loop of the shared definition sees an error, it turns toward the same target again; this is a behavior but not a policy, because it was not chosen. The retry, backoff, and circuit breaker procedure was measured in the Resilience and Reliability course; it is not repeated. The only thing measured is **what the failed tool does to the loop**. - **AQ51.** The environment, tools, tasks, and loop are **fictional**. The production stream is the same as that of the previous lesson's: twenty-four tasks run **ten times**, the budget is **6**, and the total is **240 runs**. The seeds are **20260218** and **20260219**. - **AQ52. Stop**: the loop ends at the first error; this is the baseline. **Retry**: the target does not change; at the next step the loop turns to the same target again. **Skip**: the tool that returned an error becomes **unselectable** for the rest of the run; if the draw gives it, the next tool in the list is used. - **AQ53.** There is no second tool in the set that does the same job; "moving on to another tool" means **dropping** the failed tool, and if the plan needs it, the plan cannot be completed. ## Three Policies, One Stream The block sets up the shared definition: the fictional environment, five tools, twenty-four tasks, and the loop. ```python # FICTIONAL. The measurement network's subscriber records, document set, five tools and # twenty-four tasks are defined within the lesson; no real endpoint or tool server is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "previous": previous, "current": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1) def t_read_meter(subscriber=None, **_): a = SUBSCRIBERS.get(subscriber) if a is None: return {"error": "no such subscriber"} return {"previous": a["previous"], "current": a["current"], "type": a["type"], "region": a["region"]} def t_consumption(previous=None, current=None, **_): if previous is None or current is None: return {"error": "missing field"} f = current - previous return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, type="residential", **_): if m3 is None: return {"error": "missing field"} return {"amount": _charge(m3, type)} def t_document(query=None, **_): if not query: return {"error": "missing field"} q = set(tokenize(query)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"} def t_record(subscriber=None, column=None, value=None, **_): if not (subscriber and column and value is not None): return {"error": "missing field"} return {"written": True, "subscriber": subscriber, "column": column, "value": value} TOOLS = { "read_meter": {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False}, "consumption": {"function": t_consumption, "fields": ("previous", "current"), "side_effect": False}, "tariff": {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False}, "search_docs": {"function": t_document, "fields": ("query",), "side_effect": False}, "write_record": {"function": t_record, "fields": ("subscriber", "column", "value"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"error": "no such tool"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("read_meter", "consumption", "tariff")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("read_meter", "consumption")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("search_docs",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("read_meter", "consumption", "tariff", "write_record")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["type"]) def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["approvals"] = state.get("approvals", 0) + 1 if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["result"] = s["m3"] if choice == "tariff": state["result"] = s["amount"] if choice == "search_docs": state["result"] = s["code"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} ``` The policy changes only the error branch. ```python # The recovery policy changes only the error branch of the shared loop. BUDGET, RUNS = 6, 10 RESULT_FIELD = {"consumption": "m3", "tariff": "amount", "search_docs": "code"} # tools that write the result def agent_policy(g, policy, max_steps=BUDGET, seed=SEED): r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors, banned = [], 0, 0, 0, set() seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: if r() < 0.88: break choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] while choice in banned and len(banned) < len(TOOL_NAMES): choice = TOOL_NAMES[(TOOL_NAMES.index(choice) + 1) % len(TOOL_NAMES)] if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if policy == "stop": break if policy == "skip": banned.add(choice) continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice in RESULT_FIELD: state["result"] = s[RESULT_FIELD[choice]] if choice == target: k += 1 return {"steps": step, "tokens": tok, "errors": errors, "answer": state.get("result")} FIELDS = ("steps", "tokens", "errors", "answer") print("retry is the same as the shared agent:", all(agent_policy(g, "retry") == {a: agent(g, BUDGET, SEED)[a] for a in FIELDS} for g in TASKS)) print("\nseed policy success steps errors tokens steps per correct answer") for base_seed in (20260218, 20260219): for p in ("stop", "retry", "skip"): correct, steps, tok, errors = 0, 0, 0, 0 for d in range(RUNS): for g in TASKS: s = agent_policy(g, p, BUDGET, base_seed + 7 * d) steps += s["steps"] tok += s["tokens"] errors += s["errors"] correct += s["answer"] == correct_answer(g) n = RUNS * len(TASKS) print(f"{base_seed} {p:<8} {correct / n:>8.4f} {steps:>5} {errors:>6} {tok:>9}" f" {steps / correct:>24.2f}") ``` ``` retry is the same as the shared agent: True seed policy success steps errors tokens steps per correct answer 20260218 stop 0.7500 727 53 81996 4.04 20260218 retry 0.9375 900 61 106352 4.00 20260218 skip 0.8917 907 64 108030 4.24 20260219 stop 0.7667 768 54 89180 4.17 20260219 retry 0.9542 944 66 114430 4.12 20260219 skip 0.8625 954 73 116776 4.61 ``` ## What the Policy Adds and What It Costs The first line of the output is a check: the loop we used throughout the course had an unwritten policy, and it was **retry**. **Stop** is the cheapest in all the cost columns: 727 steps, 53 tool errors, 81996 tokens. Its task success is **0.1875** below the 0.9375 of retry; since the difference between the two runs is 0.0167, the drop is ten times the band. Read per correct answer, the saving disappears: 4.04 steps for stop, 4.00 for retry, and the difference is within stop's own difference between runs (4.04 against 4.17). Stopping lowers the **spending** of steps; it does not change the **efficiency**. **Skip** is the example for the course's second claim: intuition expects the errors to decrease, and the measurement gives the opposite. The tool errors rise from 61 to **64**, and in the second run from 66 to **73**; the steps go up to 907, and the task success falls to **0.8917**. The mechanism works in two directions: the tool that gets banned has often been selected wrongly already, and the one that replaces it is wrong too, that is, the error slides onto another name; if a tool in the plan is banned, the plan ends. The loss is 0.0458 and 0.0917 in the two runs; since the widest band is 0.0292, both are above it, but the first is only one and a half times the band. ## Summary - The recovery policy is the error branch of the loop; even if it is not written, a policy runs, and here the one that runs is retry. - Stop is the cheapest (727 steps, 53 errors, 81996 tokens), lowers task success by 0.1875, and does not improve step efficiency. - Dropping the failed tool raises the tool errors to 64 and lowers task success to 0.8917. ## Course Wrap-Up | Lesson | Without scaffolding | With scaffolding | Cost | |---|---|---|---| | What Is an Agent — task success | no tools 0.0417 | loop 0.9583 (**+0.9166**) | steps from 24 to 97 | | The Agent Loop — share of right selection | depth 1: 0.8000 | depth 4: 0.5000 (**−0.3000**) | steps from 2.86 to 7.67 | | The Planner and Executor Split — task success | single loop 0.9583 | planner–executor 0.8750 (**−0.0833**) | steps from 97 to 118 | | Stopping Criteria — task success | budget 2: 0.3750 | budget 6: 1.0000 (**+0.6250**) | tokens from 4758 to 10884 | | Self-Critique and Verification — task success (24) | no rounds 21/24 | self-critique 23/24 (**+2**) | steps from 84 to 152 | | Multi-Agent Patterns — task success (24) | single agent 23/24 | two agents 23/24 (**0**) | steps from 97 to 140 | | Keeping the Human in the Loop — wrong record | approval off 3 | approval on side effects 1 (**−2**) | 7 approval points | | Tool Definition — valid call rate | name only 0.2803 | schema 0.8919 (**+0.6116**) | tool errors from 113 to 8 | | Executing Tool Calls — observation tokens | raw 770 | summarized 401 (**−369**) | task success the same: 0.9583 | | Code Execution Tools — code task success | loose limit 0.7778 | medium limit 0.3333 (**−0.4445**) | 24 stopped calls | | File System and Terminal Access — completed tasks | full authority 0.9583 | allowlist 0.8333 (**−0.1250**) | 20 stopped calls | | Web Search and Browsing — correct answer | no external knowledge 0.2083 | primary source 0.7500 (**+0.5417**) | calls from 91 to 121 | | Short-Term Memory — correct answer | unlimited window 0.9583 | window 120: 0.7083 (**−0.2500**) | tokens from 9316 to 12026 | | Long-Term Memory — steps gained (48) | no memory 0 | selective write 118 (**+118**) | 24 records | | Context Compression — total tokens | no compaction 21913 | selective 16569 (**−5344**) | 8 tasks with the field dropped | | Memory Decay — stale read share | no threshold 0.4528 | threshold 1: 0.0000 (**−0.4528**) | valid records dropped 0.9701 | | Measuring Task Success and Step Efficiency Separately — task success | toolless baseline 0.2917 | course budget 0.9583 (**+0.6666**) | per answer from 3.4286 to 4.2174 | | Testing Each Tool Separately from the Loop — defects caught (11) | untested 0 | tool level 7 (**+7**) | 0.0000 on definition defects | | Scenario-Based End-to-End Validation — defects caught (11) | tool level 7 | two layers 10 (**+3**) | residual `D11` | | Monitoring and Logging — share with cause found | result only 0.0000 | full observation 1.0000 (**+1.0000**) | tokens from 2738 to 13623 | | Cost and Latency Control — cost per step | large class 118.30 | small class 79.25 (**−39.05**) | per answer from 453.5 to 488.0 | | Error Recovery — task success (240 runs) | stop 0.7500 | retry 0.9375 (**+0.1875**) | steps from 727 to 900 | The rule of the course is written line by line in the table: **the number of an agent is the number of the whole loop, not of a single answer, and an agent whose stopping criterion is not written counts as unmeasured.** The numbers in the without-scaffolding column are not low because of a bad model; they come out low because where the loop will stop, how the tool introduces itself, what will be remembered, and what happens when a step breaks are not written. **The second claim is read from the same table too: an added step can make the work worse.** The rows that say it by name are these: Executing Tool Calls (truncation lowers 0.9583 to 0.2917 and **raises** the token to 15945), Web Search and Browsing (the secondary class lowers 0.7500 to 0.3750), Short-Term Memory (narrowing raises the token to 12026), Long-Term Memory (result only lowers 0.9167 to 0.5833), Context Compression (the tightest summarization spends 1.53 times the baseline), Multi-Agent Patterns (the second agent brings no gain), Keeping the Human in the Loop (spreading the scope raises the question to 74), Cost and Latency Control (the small class is expensive per answer), and this lesson (`skip` raises the errors to 64). In nine places the added step cost, because it was added. **Differences below the measurement band are also written by name and do not count as findings:** in What Is an Agent the baseline's single hit, in The Agent Loop the 0.0917 and 0.0160 of the neighboring depths, in The Planner and Executor Split the 0.0833 of the split, in Stopping Criteria the 0.0417 drop from six to eight, in Self-Critique and Verification the round differences, in Short-Term Memory the 0.0833 at the window of 200, in Memory Decay the gain of tightening the threshold below eight, and in this lesson the step efficiency of stop. None of them is zero, none of them is measured. In twenty-two lessons the agent was built, equipped, and measured. One thing was never asked: **every tool was defined by hand within a lesson.** The name, description, and schema of `read_meter` were written in this course and only the agent here saw them. How the same tool would be **handed** to another agent, another team, or another application, who would hold the definition, and who would break when the interface changed were not asked. The next course turns the tool definition from a dictionary inside a lesson into a **shared surface** and counts its cost.