--- title: 'AI Agents' source: 'https://academia.sh/en/courses/ai-agents' language: en updated: '2026-10-21T05:02:26+00:00' license: 'CC BY-SA 4.0' lessons: 22 outcomes: - 'Builds the agent loop by separating it into perception, reasoning, action and observation components and sets the stopping criterion with a number' - 'Measures tool definition, permission surface and memory separately and writes the cost each one adds to the loop' - 'Evaluates the agent along the axes of task success, step efficiency, trace level and failure recovery' --- # AI Agents Measuring the step on the assumption that an agent's number is the number of the whole loop, not of a single response: the task success of a single-step tool-less call and of the loop, tool selection degrading as step depth grows, what separating the planner from the executor adds and costs, the agent that has finished its work wandering when the step budget is enlarged, the wrong answer a self-critique round catches and the tokens it doubles, the communication cost a second agent brings and the field lost in the handoff, the side effect stopped per approval point, the valid call rate at four levels of tool definition, the tokens the observation format accumulates over the loop, the call a resource limit stops and the task it drops, the tasks completed as the permission surface narrows, what the source class adds to the correct answer, the information dropped when the window fills, what the write policy adds to recall accuracy, the number of tasks needing the discarded field as the compression ratio rises, the stale records falling and the lost records rising as the expiry threshold tightens, the four-state table of correct sequence and correct answer, the defect class tool-level testing cannot see, the share of failed tasks whose cause the trace level makes findable, the point where cost per step and cost per correct answer diverge, and what the recovery policy adds to success. ## Agent Architecture Building the combination of model, tool and loop and counting where the loop stops: the success of a single-step tool-less call against the loop on the same task set, the share of picking the right tool falling as step depth grows and read separately across four task kinds, a single loop compared with a planner and executor split in success, steps and tokens, a step budget raised from two steps to twelve with success first rising and then falling and that fall not counting as a finding because it stays inside the two run bands, a self-critique round with the wrong answers it catches and the steps and tokens it adds, a second agent on the same task set with the success it adds, the tokens of the handed-off context and the number of fields lost in the handoff, and a side-effect tool with the records written when approval is on and off and the wrong writes stopped. ### What Is an Agent Source: https://academia.sh/en/courses/ai-agents/what-is-an-agent Building the combination of model, tool and loop and counting the whole loop instead of a single answer: on 24 fictional tasks a single-step tool-less call gives a task success of 0.0417 while a loop with a budget of eight steps gives 0.9583, and on the second run the baseline falls to 0.0000 while the loop stays at 0.9583. The cost is a rise from 24 steps to 97 steps, from zero tool calls to 74 calls, from 1440 tokens to 11978 tokens and from zero to 8 tool errors. Read by kind, the loop finishes every task of three steps or fewer and stays at two thirds on four-step write tasks: the place where success falls is step depth. The agent does not learn; the only thing that changes over a run is the observations it holds. The Embeddings and Retrieval-Augmented Generation course measured a single question over sixteen lessons: **which information** goes into the model. Chunking, embedding, retrieval and reranking were the settings of that choice, and the cost of every setting was written in tokens. But that pipeline carried two silent assumptions. The choice was always made by **us**: code decided which chunk would be retrieved, and the model only read what was retrieved. And the pipeline was **single-pass**: the request arrived, the context was built, one answer was produced, and the job was over. This course removes both assumptions. The model itself decides which tool to call, sees the result the tool returns, and when it finds that result insufficient it tries another tool. What comes out is not a single answer but a **loop**, and its measurable surface is not that of a single answer but of the whole loop. This lesson builds that loop. **Where the loop stops** is a separate question and it stays open until the fourth lesson. - **AA1.** The code is a **simulator** and the environment is **fictional**: the fictional subscriber records of the measurement network and a fictional document collection. No real model endpoint, no real tool server and no real runtime is called. The seed is **20260218**, the second run **20260219**. - **AA2.** The environment holds **60 fictional subscribers**, five regions, two subscriber types and **5 fictional documents**. - **AA3. The tool set is five tools** and each tool sees only the **fields it declares**; fields outside the schema are filtered out, and a missing field is **the tool's own error**. - **AA4. The task set is 24 tasks** and each task is solved by a known tool sequence. There are four kinds: document (one step), consumption (two steps), charge (three steps), record (four steps). - **AA5. The share of picking the right tool falls with depth.** It is 0.80 at the first step, then falls by 0.09 at each step and levels off at 0.34. A wrong selection is a random one of the five tools. - **AA6. Observations accumulate in a state dictionary**; the input of the next tool is the output of the previous step. - **AA7.** When the job is done the agent **stops with a share of 0.88**; in the remaining share it keeps wandering and can corrupt the accumulated result. - **AA8.** `write_record` is the **only tool with a side effect**, and when approval is on each of its calls counts as an **approval point**. - **AA9. Task success is the correctness of the answer**: whether the final result the loop produced equals the known correct answer of the task. - **AA10. The baseline is a single-step, tool-less call.** The model sees only the request text. - **AA11.** The counted units are **steps, tool calls, tokens, tool errors and task success**. Time is not written. - **AA12.** Every measurement is run with **two seeds**; a difference below the run-to-run difference **counts as unmeasured**. #### Model, Tool and Loop An agent is the combination of three parts. The **model** decides what to do at each step. The **tool** is the only hand the model has reaching outside: looking at a record, doing a calculation, writing a row. The **loop** is the control flow that binds the two: the model picks a tool, the tool runs, its result comes back as an **observation**, and at the next step the model sees that observation too. If one of the three is missing there is no agent. A loop without a model is a fixed script. A tool without a loop is the **single tool call** measured in the Building Applications with Model APIs course: that lesson counted the **shape** of the call, and this course counts the **sequence** of calls. A model without tools is a single-pass call, and that is exactly the baseline of this lesson. The Reinforcement Learning and Recommender Systems course of the Machine Learning and Data Science curriculum also speaks of an agent; the agent there learns from experience and its behavior changes from run to run, while the agent here **does not learn** — the only thing that changes over a run is the observations it holds. #### Fictional Environment, Tool Set and Loop The code below builds the environment, the five tools, the twenty-four tasks and the agent loop. The same definition is used throughout the course. ```python # SIMULATOR -- FICTIONAL. No real model endpoint, tool server or runtime # is called; the environment, the tools and the tasks are all fictional. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL environment: records of the measurement network REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "previous": previous, "current": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1) def t_read_meter(subscriber=None, **_): a = SUBSCRIBERS.get(subscriber) if a is None: return {"error": "no such subscriber"} return {"previous": a["previous"], "current": a["current"], "type": a["type"], "region": a["region"]} def t_consumption(previous=None, current=None, **_): if previous is None or current is None: return {"error": "missing field"} f = current - previous return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, type="residential", **_): if m3 is None: return {"error": "missing field"} return {"amount": _charge(m3, type)} def t_document(query=None, **_): if not query: return {"error": "missing field"} q = set(tokenize(query)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"} def t_record(subscriber=None, column=None, value=None, **_): if not (subscriber and column and value is not None): return {"error": "missing field"} return {"written": True, "subscriber": subscriber, "column": column, "value": value} TOOLS = { "read_meter": {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False}, "consumption": {"function": t_consumption, "fields": ("previous", "current"), "side_effect": False}, "tariff": {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False}, "search_docs": {"function": t_document, "fields": ("query",), "side_effect": False}, "write_record": {"function": t_record, "fields": ("subscriber", "column", "value"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"error": "no such tool"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) # ---- FICTIONAL tasks: each one is solved by a known tool sequence TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("read_meter", "consumption", "tariff")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("read_meter", "consumption")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("search_docs",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("read_meter", "consumption", "tariff", "write_record")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["type"]) # ---- FICTIONAL agent: at each step it picks a tool or finishes def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["approvals"] = state.get("approvals", 0) + 1 if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["result"] = s["m3"] if choice == "tariff": state["result"] = s["amount"] if choice == "search_docs": state["result"] = s["code"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} g = TASKS[0] s = agent(g) print(f"environment : {len(SUBSCRIBERS)} subscribers , {len(DOCUMENTS)} documents ," f" {len(TOOLS)} tools , {len(TASKS)} tasks") print(f"{g['name']} ({g['kind']}) : {g['request']}") print(" trace :", " -> ".join(f"{a}{' [error]' if h else ''}" for a, h in s["trace"])) print(f" answer {s['answer']} correct {correct_answer(g)} steps {s['steps']}" f" tokens {s['tokens']} tool errors {s['errors']}") ``` ``` environment : 60 subscribers , 5 documents , 5 tools , 24 tasks G01 (document) : sayac donusunde endeks farki nasil hesaplanir trace : consumption [error] -> write_record [error] -> search_docs answer SY01 correct SY01 steps 4 tokens 462 tool errors 2 ``` A single trace shows the whole mechanics of the loop. The task is one step long: calling `search_docs` once would be enough. In the first two steps the agent reaches for other tools, and both give a **tool error**, because the state dictionary does not hold the fields those tools declare. At the third step it finds the right tool, and at the fourth step it stops. A one-step job cost **four steps** and **462 tokens**. A measurement that looks at the count of a single answer never sees these three steps. #### The Count of a Single-Step Tool-Less Call and of the Loop The baseline is tool-less: the model sees only the request text, no tool is called and no observation comes back. The same twenty-four tasks are run in two setups and with two seeds. ```python def single_step(g, seed=SEED): """BASELINE: single-step, TOOL-LESS call; no tool is called.""" r = rng(seed ^ trace_id("taban" + g["name"])) answer = (list(DOCUMENTS)[int(r() * len(DOCUMENTS))] if g["kind"] == "document" else 10 + int(r() * 900)) return {"steps": 1, "tokens": 60, "errors": 0, "answer": answer} def measure(build, seed=SEED): CALLS[0] = 0 success = steps = tok = errors = 0 for g in TASKS: s = build(g, seed) steps, tok, errors = steps + s["steps"], tok + s["tokens"], errors + s["errors"] success += s["answer"] == correct_answer(g) return success / len(TASKS), steps, CALLS[0], tok, errors def loop(g, seed): return agent(g, max_steps=8, seed=seed) print("setup success A success B step calls tokens error") for name, build in (("single-step tool-less", single_step), ("agent loop", loop)): a, b = measure(build), measure(build, SEED + 1) print(f"{name:<21} {a[0]:>9.4f} {b[0]:>10.4f} {a[1]:>6} {a[2]:>6} {a[3]:>9} {a[4]:>5}") print("loop , by kind :", " , ".join( f"{t} {sum(agent(g)['answer'] == correct_answer(g) for g in TASKS if g['kind'] == t)}" f"/{sum(g['kind'] == t for g in TASKS)}" for t in ("document", "consumption", "charge", "record"))) ``` ``` setup success A success B step calls tokens error single-step tool-less 0.0417 0.0000 24 0 1440 0 agent loop 0.9583 0.9583 97 74 11978 8 loop , by kind : document 7/7 , consumption 8/8 , charge 6/6 , record 2/3 ``` The baseline is almost zero, and why it is zero is written in the definition itself: the meter readings are in the environment, not in the request text. A tool-less call can produce a number, but that number is a guess. The **0.0417** of the first run comes from a single hit, and that hit is a guess of a document code; on the second run it drops to **0.0000**. The run-to-run difference of the baseline is 0.0417, so the baseline's only success **lies inside the measurement band** and does not count as a capability. The loop gives **0.9583** on the same task set, and this number is the same on both runs. The difference is 0.9166 and stands twenty times above the band; this gain is **measured**. Its cost sits in the same row: steps **from 24 to 97** (4.04 times), tool calls **from zero to 74**, tokens **from 1440 to 11978** (8.32 times) and tool errors **from zero to 8**. A tool-less call cannot have a tool error; if the error count is above zero, there is a loop. The last row shows where the gain ends. Every task of one, two and three steps is completed. On the four-step **record** tasks, **two** of the three tasks are correct. Because the share of picking the right tool falls with depth, the chance of missing a step builds up as the sequence grows. In a subset of three tasks the gap between 2/3 and 3/3 is not a finding by itself; but it says **on which axis** the fall lies, and that axis is step depth. #### Summary - An agent is the combination of model, tool and loop; a tool without a loop is a single tool call, and a model without tools is a single-pass call. - The environment, the five tools, the twenty-four tasks and the loop are built inside the lesson with the standard library and all of them are fictional; no real model endpoint or tool server is called. - The single-step tool-less call gives a task success of 0.0417, and 0.0000 on the second run; this single hit lies inside the measurement band and does not count as a capability. - The loop with a budget of eight steps gives 0.9583 on both runs; the gain of 0.9166 stands far above the band. - The cost is a rise from 24 to 97 in steps, from zero to 74 in tool calls, from 1440 to 11978 in tokens and from zero to 8 in tool errors. - By kind, every task of one, two and three steps is completed, and two thirds of the four-step write tasks are completed. #### Next Step In this lesson the loop was measured as a box: the task going in, the answer coming out and the total cost in between. The inside of the box showed up in a single trace, and that trace carried two wrong tool selections. The last row by kind points the same way: as the job grows longer, success falls. The next lesson opens the inside of the loop step by step — it writes which part each step corresponds to as perception, reasoning, action and observation, and counts the share of picking the right tool **by step depth**. The same number is read separately across the four task kinds, because the kinds have different sequence lengths and that is exactly where the fall hides in a single average. ### The Agent Loop Source: https://academia.sh/en/courses/ai-agents/agent-loop Opening one step of the loop into perception, reasoning, action and observation and counting the share of picking the right tool by step depth: on 24 fictional tasks the share is 0.8000 at depth 1 and 0.5000 at depth 4; on the second run 0.7742 and 0.5000. The fall of 0.3000 between them stands far above the widest run-to-run band (0.0644) and is measured, but the differences of 0.0917 and 0.0160 between neighboring depths stay inside the band and count as unmeasured. As depth grows the measurement itself thins out: the number of selections falls from 30 to 6. Read across four task kinds, the success of tasks of one, two and three steps is 1.0000 on both runs and 0.6667 on four-step write tasks; steps per task rise from 2.86 to 7.67. A wrong selection does not only spend a step, it can also overwrite the accumulated result. The previous lesson measured the loop as a box: twenty-four tasks going in, the answers coming out and the total cost in between. The inside of the box showed up in a single trace only, that trace held two wrong tool selections, and the last row by kind said that success falls on four-step tasks. Where the fall comes from stayed open. This lesson opens the box. It first writes the phases a step goes through, then counts one number **by step depth**: whether the agent was able to pick the **right tool** at that step. The baseline of this lesson is not the scaffolding-free loop but **the first step of the same loop**; as depth grows, what the same measure becomes is counted. - **AA13.** The environment, the tool set, the task set and the loop are the same definition as in the previous lesson and are **fictional**. The seed is **20260218**, the second run **20260219**. - **AA14. A step is four phases:** reading the fields in the state (**perception**), determining the next target (**reasoning**), calling a tool (**action**) and writing the returned result into the state (**observation**). - **AA15. Depth is which step of the task's sequence one is at** — not the number of steps spent. A target missed three times consumes three steps but does not change the depth. - **AA16. The share of picking the right tool** is how many of the selections made at that depth were the target tool. Because a wrong selection is a random one of five tools, it **can also land on the target by chance**; the share the mechanism predicts is therefore `selection(d) + (1 - selection(d)) / 5`. - **AA17. Steps after the job is done do not count as selections**; they have no target. - **AA18. A wasted call** is a selection that does not hit the target. It spends a step, spends tokens and can write an **unwanted field** into the state. - **AA19.** The step budget is fixed at **eight** in this lesson; the budget is the subject of the fourth lesson. - **AA20.** As depth grows, fewer tasks reach it; **the sample thins out too** and is written on every row. - **AA21.** A difference below the run-to-run difference **counts as unmeasured**. #### Four Phases and a Breakdown of One Task Every step of the loop goes through the same four phases. **Perception** is what the state dictionary holds at that moment: which fields are filled, which observations have accumulated. **Reasoning** is deciding what the next target is. **Action** is calling a tool. **Observation** is writing the fields the tool returned into the state — or it is a **tool error**, because the call did not find the fields the tool declares. The code below builds the definition and prints the breakdown of a four-step write task. ```python # SIMULATOR -- FICTIONAL. No real model endpoint, tool server or runtime # is called; the environment, the tools and the tasks are all fictional. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL environment: records of the measurement network REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "previous": previous, "current": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1) def t_read_meter(subscriber=None, **_): a = SUBSCRIBERS.get(subscriber) if a is None: return {"error": "no such subscriber"} return {"previous": a["previous"], "current": a["current"], "type": a["type"], "region": a["region"]} def t_consumption(previous=None, current=None, **_): if previous is None or current is None: return {"error": "missing field"} f = current - previous return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, type="residential", **_): if m3 is None: return {"error": "missing field"} return {"amount": _charge(m3, type)} def t_document(query=None, **_): if not query: return {"error": "missing field"} q = set(tokenize(query)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"} def t_record(subscriber=None, column=None, value=None, **_): if not (subscriber and column and value is not None): return {"error": "missing field"} return {"written": True, "subscriber": subscriber, "column": column, "value": value} TOOLS = { "read_meter": {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False}, "consumption": {"function": t_consumption, "fields": ("previous", "current"), "side_effect": False}, "tariff": {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False}, "search_docs": {"function": t_document, "fields": ("query",), "side_effect": False}, "write_record": {"function": t_record, "fields": ("subscriber", "column", "value"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"error": "no such tool"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) # ---- FICTIONAL tasks: each one is solved by a known tool sequence TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("read_meter", "consumption", "tariff")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("read_meter", "consumption")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("search_docs",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("read_meter", "consumption", "tariff", "write_record")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["type"]) # ---- FICTIONAL agent: at each step it picks a tool or finishes def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["approvals"] = state.get("approvals", 0) + 1 if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["result"] = s["m3"] if choice == "tariff": state["result"] = s["amount"] if choice == "search_docs": state["result"] = s["code"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} def read_trace(g, seed=SEED): """Rereads the trace against the task's sequence: at each step the target of reasoning, the tool chosen as action and what the observation brought.""" s = agent(g, seed=seed) k, pairs, lines = 0, [], [] for i, (choice, failed) in enumerate(s["trace"], 1): target = g["sequence"][k] if k < len(g["sequence"]) else "none , must stop" if k < len(g["sequence"]): pairs.append((k + 1, choice == target)) lines.append(f" {i}. depth {min(k + 1, len(g['sequence'])):<2}" f" target {target:<17} action {choice:<12}" f" observation {'tool error' if failed else 'fields arrived'}") if choice == target and not failed: k += 1 return s, pairs, lines g = [x for x in TASKS if x["name"] == "G23"][0] s, _, lines = read_trace(g) print(f"{g['name']} ({g['kind']}) : {g['request']}") print("\n".join(lines)) print(f" step {s['steps']} answer {s['answer']} correct answer {correct_answer(g)}") ``` ``` G23 (record) : A006 abonesinin bedelini hesapla ve kayda yaz 1. depth 1 target read_meter action read_meter observation fields arrived 2. depth 2 target consumption action consumption observation fields arrived 3. depth 3 target tariff action search_docs observation fields arrived 4. depth 3 target tariff action consumption observation fields arrived 5. depth 3 target tariff action tariff observation fields arrived 6. depth 4 target write_record action tariff observation fields arrived 7. depth 4 target write_record action search_docs observation fields arrived 8. depth 4 target write_record action write_record observation fields arrived step 8 answer TR01 correct answer 1132 ``` This breakdown shows **two separate** costs of a wrong selection. The `search_docs` and `consumption` calls at the third and fourth steps miss the target; neither gives an error, because the fields they expect are in the state. A step is spent and the depth does not advance. The seventh step is heavier: `search_docs` returns successfully and **overwrites the accumulated result**; the state now holds a document code instead of 1132. At the eighth step the agent finally hits the target, but `write_record` puts that corrupted value into the record. A wasted call that returns successfully does not only spend a step, it **corrupts the state**. #### The Share of Picking the Right Tool as Depth Grows The same number is added up over the whole task set, depth by depth. The share the mechanism predicts is written next to it, because a wrong selection is a random tool and can also land on the target by chance. ```python print("depth expected A: picks right share B: picks right share") for d in (1, 2, 3, 4): row = [f"{d:>5} {selection(d) + (1 - selection(d)) / 5:>12.4f}"] for seed in (SEED, SEED + 1): n = k = 0 for task in TASKS: for depth, ok in read_trace(task, seed)[1]: if depth == d: n, k = n + 1, k + ok row.append(f"{n:>10} {k:>6} {k / n:>6.4f}") print(" ".join(row)) print() print("kind seq tasks A: steps/task wasted errors success" " B: steps/task wasted errors success") for kind, length in (("document", 1), ("consumption", 2), ("charge", 3), ("record", 4)): subset = [x for x in TASKS if x["kind"] == kind] row = [f"{kind:<11} {length:>4} {len(subset):>6}"] for seed in (SEED, SEED + 1): steps = wasted = errors = hits = 0 for task in subset: res, pairs, _ = read_trace(task, seed) steps, errors = steps + res["steps"], errors + res["errors"] wasted += sum(1 for _, ok in pairs if not ok) hits += res["answer"] == correct_answer(task) row.append(f"{steps / len(subset):>13.2f} {wasted:>6} {errors:>6} {hits / len(subset):>8.4f}") print(" ".join(row)) ``` ``` depth expected A: picks right share B: picks right share 1 0.8400 30 24 0.8000 31 24 0.7742 2 0.7680 24 17 0.7083 22 17 0.7727 3 0.6960 13 9 0.6923 12 9 0.7500 4 0.6240 6 3 0.5000 6 3 0.5000 kind seq tasks A: steps/task wasted errors success B: steps/task wasted errors success document 1 7 2.86 5 5 1.0000 2.86 3 2 1.0000 consumption 2 8 3.50 4 3 1.0000 3.62 5 1 1.0000 charge 3 6 4.33 2 0 1.0000 5.00 6 3 1.0000 record 4 3 7.67 9 0 0.6667 6.33 4 1 0.6667 ``` The first table says two things at once, and they must be read separately. The end-to-end difference is **measured**: the share is **0.8000** at depth 1 and **0.5000** at depth 4; on the second run **0.7742** and **0.5000**. The fall is 0.3000 and 0.2742. The run-to-run difference of the same depth is **0.0644** at its widest (depth 2), so the end-to-end fall is more than four times the band. As the plan grows longer, the selection deteriorates. The differences between neighboring depths tell a different story. On the first run the fall from 1 to 2 is 0.0917 and from 2 to 3 is 0.0160; on the second run the first is 0.0015 and the second is 0.0227. All of these are inside the band or at its edge and **count as unmeasured**. Only the move from 3 to 4 exceeds the band on both runs: 0.1923 and 0.2500. A measurement that wants to read the share of single steps needs a larger task set. The reason is written in the selection columns on the right. There are **30** selections at depth 1 and only **6** at depth 4. The number of tasks with a four-step sequence is three, and the number of runs that reach there is even smaller. **As depth grows, the measurement itself thins out**: the place where the agent is weakest is the place where we have the fewest numbers. #### Reading Across Four Task Kinds The second table shows the same fall by kind. Steps per task grow with sequence length: **2.86**, **3.50**, **4.33**, **7.67**. The minimum steps needed are 2, 3, 4 and 5 respectively — one step always goes to stopping. On the four-step kind the realized steps are **1.53 times** the minimum needed, and this ratio falls to 1.27 on the second run; the difference between the two cannot be measured in a subset of three tasks. What can be measured is success. The one-, two- and three-step kinds give **1.0000** on both runs and the four-step kind gives **0.6667** on both runs. The same agent, the same tools, the same budget; the only thing that changes is the length of the sequence. The tool error column, on the other hand, swings from 5 to 2 and from 0 to 3 between the two runs: at this scale error counts are noise. #### Summary - An agent step is four phases: reading the state, determining the target, calling the tool and writing the returned result into the state. - A wrong selection pays two costs: the step spent and the unwanted field written into the state; a successful wrong call can overwrite the accumulated result. - The share of picking the right tool is 0.8000 at depth 1 and 0.5000 at depth 4; on the second run 0.7742 and 0.5000. The end-to-end fall is more than four times the widest run-to-run band and is measured. - The differences of 0.0917 and 0.0160 between neighboring depths are inside the band and count as unmeasured; only the move from the third depth to the fourth exceeds the band on both runs. - As depth grows the sample thins out: the number of selections falls from 30 to 6, so the place where the agent is weakest is the place where we have the fewest numbers. - By kind, steps per task rise from 2.86 to 7.67; success is 1.0000 on the three short kinds and 0.6667 on both runs on the four-step kind. #### Next Step In this lesson a single loop did all the work itself: it determined the target and it called the tool. The halving of the share of hitting the target at the fourth step of the sequence is the cost of that combination, because the same call has to carry both the whole plan and the next action. The first solution that comes to mind is to split the job: let one side write a plan and let the other side only carry out the steps of that plan. The next lesson builds this split and asks two questions together — whether the split raises success, and how much the split job costs in steps and tokens. ### The Planner and Executor Split Source: https://academia.sh/en/courses/ai-agents/planner-and-executor-split What separating the side that decides from the side that runs adds and what it costs: on 24 fictional tasks the single loop gives a task success of 0.9583, 97 steps, 74 tool calls and 11978 tokens; once planner and executor are separated, success becomes 0.8750, steps 118, calls 71 and tokens 10345. The difference in success equals the split arrangement's own run-to-run difference and counts as unmeasured. The extra steps are exactly one planning step per task. Tokens fall by more than thirteen percent on both runs, because the executor carries the plan, not the reasoning trace. 0.8868 of the plan's positions are right and eighteen of the twenty-four plans are right from start to finish; since a wrong position is executed blindly, correction starts only after the plan runs out. The previous lesson measured how a single loop deteriorates with depth: at the fourth step of the sequence the share of picking the right tool was cut in half. One reason is structural — the same call carries both the plan of the whole job and the single next action. The first arrangement that comes to mind is to split the job in two. Let one side put out the **plan** of the job in one go, and let the other side **execute** the steps of that plan and not pick tools by itself. This lesson builds the split and asks two questions together: does the split raise success, and what does it make one pay. The baseline is the scaffolding-free loop — the single loop of the previous lesson. - **AA22.** The environment, the tool set, the task set and the single loop are the same definition as in the previous lessons and are **fictional**. The seed is **20260218**, the second run **20260219**. - **AA23. The split arrangement is two parts:** the **planner** writes the whole plan in a single call, and the **executor** runs the plan in order and does not pick tools by itself. - **AA24. The planner sees no observation.** It picks all positions at the same time, with the same information; so the depth penalty does not apply to it and it picks every position with the flat share `selection(1)`. - **AA25. Planning is a step** and is deducted from the budget. Both setups are run with the same budget of **eight steps**. - **AA26. The executor runs the plan blindly**: a wrong position is executed too. - **AA27.** If the plan ends and the job is not done, the remaining positions are tried **by the rule of the single loop**; correction starts only after the plan runs out. - **AA28. Tokens are counted separately in the two setups.** A step of the single loop is `60 + 22 x trace length`, the planner's call is `60 + 25 x plan length`, a step of the executor is `40 + 10 x trace length`; every observation is 30 tokens. The executor carries the **plan**, not the reasoning trace. - **AA29.** Plan length is the sequence length of the task; the planner does not know the sequence, it knows its length. - **AA30.** The counted units are **success, steps, tool calls, tokens and tool errors**; a difference below the run-to-run difference **counts as unmeasured**. #### The Split: One Plan, One Execution In the split arrangement the decision is made once. The planner reads the request and writes a sequence such as `read_meter -> consumption -> tariff`. The executor runs that sequence in order, and because it makes no selection its step is cheap: it carries the plan and the accumulated state, not the reasoning trace. The cost of the split is written in the same sentence — because the plan is written **before the observation**, a wrong position is not understood to be wrong until it has been executed. ```python # SIMULATOR -- FICTIONAL. No real model endpoint, tool server or runtime # is called; the environment, the tools and the tasks are all fictional. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL environment: records of the measurement network REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "previous": previous, "current": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1) def t_read_meter(subscriber=None, **_): a = SUBSCRIBERS.get(subscriber) if a is None: return {"error": "no such subscriber"} return {"previous": a["previous"], "current": a["current"], "type": a["type"], "region": a["region"]} def t_consumption(previous=None, current=None, **_): if previous is None or current is None: return {"error": "missing field"} f = current - previous return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, type="residential", **_): if m3 is None: return {"error": "missing field"} return {"amount": _charge(m3, type)} def t_document(query=None, **_): if not query: return {"error": "missing field"} q = set(tokenize(query)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"} def t_record(subscriber=None, column=None, value=None, **_): if not (subscriber and column and value is not None): return {"error": "missing field"} return {"written": True, "subscriber": subscriber, "column": column, "value": value} TOOLS = { "read_meter": {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False}, "consumption": {"function": t_consumption, "fields": ("previous", "current"), "side_effect": False}, "tariff": {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False}, "search_docs": {"function": t_document, "fields": ("query",), "side_effect": False}, "write_record": {"function": t_record, "fields": ("subscriber", "column", "value"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"error": "no such tool"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) # ---- FICTIONAL tasks: each one is solved by a known tool sequence TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("read_meter", "consumption", "tariff")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("read_meter", "consumption")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("search_docs",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("read_meter", "consumption", "tariff", "write_record")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["type"]) # ---- FICTIONAL agent: at each step it picks a tool or finishes def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["approvals"] = state.get("approvals", 0) + 1 if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["result"] = s["m3"] if choice == "tariff": state["result"] = s["amount"] if choice == "search_docs": state["result"] = s["code"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} def plan_and_execute(g, max_steps=8, seed=SEED): """SPLIT arrangement: the PLANNER writes the whole plan in a single call; since it sees no observation it picks every position with the flat selection(1) share. The EXECUTOR runs the plan in order and makes no selection; if the plan ends and the job is not done, the remaining positions are tried by the single loop's rule.""" r = rng(seed ^ trace_id("plan" + g["name"])) seq = g["sequence"] plan = [d if r() < selection(1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] for d in seq] state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, errors, k = [], 1, 0, 0 # planning is a step tok = 60 + 25 * len(plan) # the planner's call while step < max_steps: step += 1 tok += 40 + 10 * len(trace) # the executor carries the plan, not the trace target = seq[k] if k < len(seq) else None if target is None: if r() < 0.88: break choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] elif len(trace) < len(plan): choice = plan[len(trace)] # the executor follows the plan's order else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) for a, b in (("consumption", "m3"), ("tariff", "amount"), ("search_docs", "code")): if choice == a: state["result"] = s[b] if choice == target: k += 1 return {"steps": step, "trace": trace, "tokens": tok, "errors": errors, "plan": plan, "answer": state.get("result")} g = [x for x in TASKS if x["name"] == "G05"][0] s = plan_and_execute(g) print(f"{g['name']} ({g['kind']}) correct sequence :", " -> ".join(g["sequence"])) print(" plan :", " -> ".join(s["plan"])) print(" executed :", " -> ".join(f"{a}{' [error]' if h else ''}" for a, h in s["trace"])) print(f" steps {s['steps']} tokens {s['tokens']} tool errors {s['errors']}" f" answer {s['answer']} correct {correct_answer(g)}") ``` ``` G05 (charge) correct sequence : read_meter -> consumption -> tariff plan : tariff -> consumption -> tariff executed : tariff [error] -> consumption [error] -> tariff [error] -> read_meter -> consumption -> tariff steps 8 tokens 805 tool errors 3 answer 550 correct 550 ``` The first position of the plan is wrong, and this single mistake wastes the whole plan. `tariff` gives an error because it finds no `m3` in the state, `consumption` gives an error because the meter readings were not read, and the third position is `tariff` again and an error again. Since the executor is bound to the plan, it has to run all three; correction starts only when the plan runs out. The correct answer comes out in the end, but only after **three tool errors** were paid. #### What the Split Adds and What It Costs The same twenty-four tasks are run in two setups, with two seeds and with the same budget of eight steps. ```python def measure(build, seed=SEED): CALLS[0] = 0 success = steps = tok = errors = 0 for t in TASKS: y = build(t, seed=seed) steps, tok, errors = steps + y["steps"], tok + y["tokens"], errors + y["errors"] success += y["answer"] == correct_answer(t) return success / len(TASKS), steps, CALLS[0], tok, errors print("setup run success step calls tokens error") for name, build in (("single loop", agent), ("planner + executor", plan_and_execute)): for label, seed in (("A", SEED), ("B", SEED + 1)): m = measure(build, seed) print(f"{name:<22} {label:>5} {m[0]:>8.4f} {m[1]:>6} {m[2]:>6} {m[3]:>9} {m[4]:>5}") total = sum(len(t["sequence"]) for t in TASKS) for label, seed in (("A", SEED), ("B", SEED + 1)): plans = [plan_and_execute(t, seed=seed)["plan"] for t in TASKS] right = sum(a == b for pl, t in zip(plans, TASKS) for a, b in zip(pl, t["sequence"])) whole = sum(list(pl) == list(t["sequence"]) for pl, t in zip(plans, TASKS)) print(f"plan {label} : right positions {right}/{total} = {right / total:.4f} ;" f" fully right plans {whole}/{len(TASKS)}") ``` ``` setup run success step calls tokens error single loop A 0.9583 97 74 11978 8 single loop B 0.9583 98 74 11994 7 planner + executor A 0.8750 118 71 10345 10 planner + executor B 0.9583 116 70 10135 12 plan A : right positions 47/53 = 0.8868 ; fully right plans 18/24 plan B : right positions 47/53 = 0.8868 ; fully right plans 18/24 ``` The success column **yields no finding**. The single loop is 0.9583 on both runs; the split arrangement is 0.8750 and 0.9583. The split arrangement's own run-to-run difference is **0.0833**; the difference between the two setups is exactly 0.0833 on the first run and **zero** on the second. The difference is inside the band, and **the effect of the split on success counts as unmeasured**. A difference that can be seen with the eye is not a measured difference. The difference in the step column is measured, and where it comes from can be calculated exactly. The split arrangement adds **one planning step** to every task, that is twenty-four steps; 97 plus 24 is 121, and the realized figure is 118. The counterpart of the three missing steps is in the neighboring column: tool calls fall **from 74 to 71**. The plan protects the executor from a number of wasted calls, but planning itself takes back eight times that gain. The token column is where the split **adds**: against 11978 and 11994 stand 10345 and 10135, a fall of 13.6 percent and 15.5 percent. The two setups' own bands are 16 and 210 tokens; the difference stands far above the band and is **measured**. The reason is in the definition — in the single loop every step carries the growing reasoning trace again, while in the executor a step carries a fixed plan. Tool errors go the other way: against 8 and 7 stand 10 and 12. The direction is the same on both runs and its mechanism is clear; a blindly executed wrong position calls a tool whose fields are missing from the state. But with bands of 1 and 2, the difference on the first run is 2: **we know the direction from the mechanism, we cannot measure the size at this scale.** The last two lines give the limit of the split. **0.8868** of the plan's positions are right, but only **18** of the twenty-four plans are right from start to finish. As the plan grows longer, the chance of all of it being right falls, and a single wrong position can waste the whole plan as in G05. Splitting gathers the decision at a single point; it also gathers the cost of an error at that point at a single point. #### Summary - In the split arrangement the planner writes the whole plan before the observation and the executor runs it in order and blindly; correction starts only after the plan runs out. - Success is 0.9583 and 0.9583 in the single loop and 0.8750 and 0.9583 in the split arrangement; the difference equals the split arrangement's own run-to-run difference and counts as unmeasured. - The step the split adds is exactly one planning step per task: 24 is added to 97 steps, and since tool calls fall from 74 to 71, the total stays at 118. - Tokens fall from 11978 to 10345 and from 11994 to 10135; the fall stands far above the band, because the executor carries the plan, not the reasoning trace. - Tool errors are high in the split arrangement on both runs (10 and 12 against 8 and 7); the direction is clear from the mechanism, the size cannot be measured at this scale. - 0.8868 of the plan's positions are right, but the plans that are fully right are eighteen of twenty-four; a single wrong position can waste the whole plan. #### Next Step Both setups stumble in the same place, and that place has not been asked about for three lessons: **where will the loop stop**. Eight steps was said and eight steps was written, but that number was a decision and its reason was not written. After the plan ran out, G05 needed three more steps to correct itself; with a budget of six it would have given a wrong answer. On the other hand, an agent that has finished its job and cannot manage to stop can corrupt the correct result it holds. The next lesson sweeps the step budget over the values two, four, six, eight and twelve, and writes the stopping criterion as the rule of the course. ### Stopping Criteria Source: https://academia.sh/en/courses/ai-agents/stopping-criteria Writing where the loop stops as the rule of the course and sweeping the step budget over the values two, four, six, eight and twelve: on 24 fictional tasks task success rises from 0.3750 to 1.0000 and then flattens at 0.9583; on the second run it is 0.4167, 0.8333, 0.9583, 0.9583, 0.9583. The fall from the sixth step to the eighth is 0.0417 and the run-to-run difference at the sixth step is also 0.0417, so the fall is inside the measurement band and cannot be declared a finding; its mechanism is shown on the trace of a single task and its effect counts as unmeasured. When the budget goes from six to twelve, success does not change while tokens rise from 10884 to 12214. Stopping by budget is not a stopping criterion but a safety limit: at the sixth step three tasks stop because the budget ran out. All three previous lessons held the same number fixed: the step budget was eight. That was a decision, its reason was not written, and all the results hang on it. In the split arrangement G05 needed three more steps to correct itself after its plan ran out; with a budget of six it would have given a wrong answer. The reverse is also true: an agent that has finished its job and cannot manage to stop can corrupt the correct result it holds. This lesson measures that decision. The step budget is swept over five values, and at each value the cost is written next to success. The lesson also sets the rule of the course: **the number of an agent is the number of the whole loop, not of a single answer, and an agent whose stopping condition is not written counts as unmeasured.** The baseline is the narrowest budget — a loop of two steps. - **AA31.** The environment, the tool set, the task set and the loop are the same definition as in the previous lessons and are **fictional**. The seed is **20260218**, the second run **20260219**. - **AA32. The stopping criterion is a part of the loop**, not a safety put on from outside. This simulator carries two criteria: **the decision that the job is done** and **the step budget**. - **AA33. The decision that the job is done is uncertain.** When no target is left in the sequence the agent stops with a **share of 0.88**; with the remaining share it calls one more tool. - **AA34. A runaway loop** is a loop that keeps taking steps although it has no target left. Every step it takes is counted. - **AA35. The step budget is not a stopping criterion but a hard limit.** When the budget runs out the loop is cut off midway, and whatever value it holds is the answer. - **AA36. The swept values are 2, 4, 6, 8 and 12**; everything else is fixed. - **AA37.** The counted units are **success, steps, tool calls, tokens, tool errors, runs that end by their own decision, and wandering steps**. - **AA38. The task set is 24 tasks**, so the smallest step of success is **0.0417**. A difference smaller than this cannot be represented in this set. - **AA39.** A difference below the run-to-run difference **counts as unmeasured**; this lesson applies the rule to its own result as well. - **AA40.** Time is not written. #### What a Stopping Criterion Is When a loop will stop is as much a design decision as what it will do. There are two things to tell apart. **The decision that the job is done** is the agent's own decision: no target is left in line, there is a result in hand, stop. **The step budget** is a hard limit put on from outside and is not a decision; it cuts the loop wherever it happens to be. The second cannot take the place of the first, because a loop that is cut off hands over the value it holds as the answer, and the correctness of that value has not been looked at. The classes of criteria are not limited to these: a repeated step, a token budget and an external cancellation are also stopping criteria. This simulator carries only the first two, and only their numbers are printed in this lesson. ```python # SIMULATOR -- FICTIONAL. No real model endpoint, tool server or runtime # is called; the environment, the tools and the tasks are all fictional. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL environment: records of the measurement network REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "previous": previous, "current": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1) def t_read_meter(subscriber=None, **_): a = SUBSCRIBERS.get(subscriber) if a is None: return {"error": "no such subscriber"} return {"previous": a["previous"], "current": a["current"], "type": a["type"], "region": a["region"]} def t_consumption(previous=None, current=None, **_): if previous is None or current is None: return {"error": "missing field"} f = current - previous return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, type="residential", **_): if m3 is None: return {"error": "missing field"} return {"amount": _charge(m3, type)} def t_document(query=None, **_): if not query: return {"error": "missing field"} q = set(tokenize(query)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"} def t_record(subscriber=None, column=None, value=None, **_): if not (subscriber and column and value is not None): return {"error": "missing field"} return {"written": True, "subscriber": subscriber, "column": column, "value": value} TOOLS = { "read_meter": {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False}, "consumption": {"function": t_consumption, "fields": ("previous", "current"), "side_effect": False}, "tariff": {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False}, "search_docs": {"function": t_document, "fields": ("query",), "side_effect": False}, "write_record": {"function": t_record, "fields": ("subscriber", "column", "value"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"error": "no such tool"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) # ---- FICTIONAL tasks: each one is solved by a known tool sequence TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("read_meter", "consumption", "tariff")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("read_meter", "consumption")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("search_docs",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("read_meter", "consumption", "tariff", "write_record")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["type"]) # ---- FICTIONAL agent: at each step it picks a tool or finishes def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["approvals"] = state.get("approvals", 0) + 1 if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["result"] = s["m3"] if choice == "tariff": state["result"] = s["amount"] if choice == "search_docs": state["result"] = s["code"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} def wandering(g, s): """Number of steps taken after the job is done: tool calls made although no target is left. Every step for which no decision to stop was made is written here.""" k = n = 0 for choice, failed in s["trace"]: if k >= len(g["sequence"]): n += 1 elif choice == g["sequence"][k] and not failed: k += 1 return n def sweep(b, seed=SEED): CALLS[0] = 0 success = step = tok = error = done = wander = 0 for x in TASKS: y = agent(x, max_steps=b, seed=seed) step, tok, error = step + y["steps"], tok + y["tokens"], error + y["errors"] success += y["answer"] == correct_answer(x) done += y["done"] wander += wandering(x, y) return success / len(TASKS), step, CALLS[0], tok, error, done, wander for label, seed in (("A", SEED), ("B", SEED + 1)): print(f"run {label} budget success step calls tokens error done wandering") for b in (2, 4, 6, 8, 12): z = sweep(b, seed) print(f" {b:>6} {z[0]:>8.4f} {z[1]:>6} {z[2]:>6} {z[3]:>9}" f" {z[4]:>5} {z[5]:>6} {z[6]:>9}") ``` ``` run A budget success step calls tokens error done wandering 2 0.3750 48 45 4758 8 3 1 4 0.8750 84 64 9402 8 20 1 6 1.0000 92 71 10884 8 21 1 8 0.9583 97 74 11978 8 23 1 12 0.9583 98 74 12214 8 24 1 run B budget success step calls tokens error done wandering 2 0.4167 48 45 4758 7 3 2 4 0.8333 83 67 9366 7 16 3 6 0.9583 96 74 11610 7 22 3 8 0.9583 98 74 11994 7 24 3 12 0.9583 98 74 11994 7 24 3 ``` The first three rows are beyond dispute. The budget of two steps gives sequences of three and four steps no chance at all; success can come only from the short tasks and stays at **0.3750** and **0.4167**. At the fourth step it is 0.8750 and 0.8333, at the sixth step **1.0000** and **0.9583**. The gain from two to six is 0.6250 and 0.5416; since the widest run-to-run difference is **0.0417**, this gain is more than ten times the band and is **measured**. The gain from four to six is 0.1250 on both runs, and that too is above the band. The `done` column changes the nature of this gain. With the budget of two steps only **three** of the twenty-four runs stop by their own decision; the other twenty-one are cut off because the budget ran out. Even at the sixth step three runs end by being cut off. A run that is cut off has not produced an answer; it has handed over the value it holds. **The budget is not a stopping criterion but a limit of last resort in the absence of one.** #### From the Sixth Step to the Eighth: The Mechanism Exists, the Measure Does Not The striking place in the table is the move from the sixth row to the eighth. On the first run success falls **from 1.0000 to 0.9583**. A single task is responsible for the fall, and its trace can be read. ```python g = [x for x in TASKS if x["name"] == "G23"][0] print(f"{g['name']} ({g['kind']}) sequence :", " , ".join(g["sequence"]), "| correct answer", correct_answer(g)) for b in (6, 8, 12): s = agent(g, max_steps=b) print(f" budget {b:>2} step {s['steps']} done {s['done']} answer {s['answer']}") print(" trace :", " , ".join(a for a, _ in s["trace"])) ``` ``` G23 (record) sequence : read_meter , consumption , tariff , write_record | correct answer 1132 budget 6 step 6 done False answer 1132 trace : read_meter , consumption , search_docs , consumption , tariff , tariff budget 8 step 8 done False answer TR01 trace : read_meter , consumption , search_docs , consumption , tariff , tariff , search_docs , write_record budget 12 step 9 done True answer TR01 trace : read_meter , consumption , search_docs , consumption , tariff , tariff , search_docs , write_record ``` With the budget of six steps G23 gives the right answer, but **not by its own decision**: the budget runs out at the sixth step and the value in the state at that moment is 1132, which is correct. The budget of eight steps allows two more steps. At the seventh step the agent picks `search_docs`; this choice is wrong but **gives no error**, because the field the tool asks for is in the state. The call, returning successfully, writes a document code over the accumulated result. At the eighth step the target is finally hit and `write_record` puts that corrupted value into the record. The budget of twelve steps also makes the decision to stop at the ninth step, but there is nothing left to correct. The mechanism is clear and can be written in one sentence: **once the correct result has been written to the state, every extra step is a chance to corrupt it.** The second channel of the loop looks the same way — when the job is done the agent does not stop with a share of 0.12 and calls one more tool; the number of wandering steps in these runs is 1 and 3. On the other hand, **this fall is not a finding**. The numbers the two runs give at the sixth step are 1.0000 and 0.9583, so the run-to-run difference at that point is **0.0417**. The fall from six to eight is also exactly 0.0417, and on the second run it is **zero**. The difference stays inside the measurement band: **the mechanism can be shown, its effect counts as unmeasured.** This is a limit imposed by scale. In a set of twenty-four tasks a task is worth 0.0417, and the band is also one task wide, so an effect of one task can never rise above the band in this set. The reader who wants to measure the effect has two ways. The first is **enlarging the task set**: if the set grows to several times its size, the resolution increases. The second is **sweeping the wandering share as a setting**; if 0.12 stops being a single point and turns into an axis, how success changes along that axis can be read without depending on the difference of a single task. What is measured is the cost. When the budget goes from six to twelve, success does not change on the first run, whereas tokens rise **from 10884 to 12214**, steps from 92 to 98 and tool calls from 71 to 74. The move from eight to twelve leaves success fixed on both runs. **The gain of enlarging the budget could not be measured; its cost was measured.** #### Summary - The stopping criterion is a part of the loop; the decision that the job is done is the agent's own decision, the step budget is a hard limit put on from outside, and it cannot take the place of the first. - Success is 0.3750 and 0.4167 at two steps and 1.0000 and 0.9583 at six steps; the gain from two to six is more than ten times the widest run-to-run difference and is measured. - With the budget of two steps only three of the twenty-four runs stop by their own decision; even at six steps three runs are cut off because the budget ran out. - The fall from six to eight is 0.0417 and the run-to-run difference at the sixth step is also 0.0417; the mechanism can be shown on the trace of a single task, but its effect counts as unmeasured. - Measuring the effect requires enlarging the task set or sweeping the wandering share on its own axis. - When the budget goes from six to twelve, success does not change while tokens rise from 10884 to 12214: the gain could not be measured, the cost was measured. #### Next Step In four lessons the loop was built, opened up step by step, split in two, and where it stops was written. All these measurements had a common support, and that support was not questioned: **we** knew the correct answer. The task set carried the correct answer of every task, and we calculated success by comparing against that list. When the agent itself stopped, it never looked once at the value in its hand; G23 put a corrupted result into the record and did not see it as a problem. The next lesson enters this gap: if the agent adds a round that checks its own output, what does it catch, and how many steps and how many tokens does that round cost. ### Self-Critique and Verification Source: https://academia.sh/en/courses/ai-agents/self-critique-and-verification What it adds and what it costs for an agent to look at its own trace after it stops: when three observable signs are read on twenty-four fictional tasks, the round flags every wrong answer at narrow budgets but cannot see at all a wrong answer with a clean trace at eight steps. At a budget of four steps success rises from 21/24 to 23/24, in return for steps going from 84 to 152 and tokens from 9402 to 17052. When the same steps are given to a single longer run, the eight-step budget gives the same 23/24 with 97 steps and 11978 tokens. At six steps the round lowers success from 24/24 to 23/24; at eight steps it changes nothing, and six of the seven flags are unnecessary reruns. The run-to-run difference is one task, so the difference at six and eight steps counts as unmeasured. In the previous four lessons the loop was built and where it stops was written. The agent picks a tool, writes the returned observation to its state, takes the next step by looking at it; it stops when its job is done or the budget is full. When it closes, an answer is left in hand. The one thing that was not asked in four lessons is this: **is that answer correct.** When the agent stops, it does not turn back and look at its own trace. This lesson adds that missing round and measures two things: what the round **catches** and what the round **costs**. Running the same prompt several times and voting on the answers is a separate arrangement and was measured in the Prompt Engineering course; here there are no several answers to vote on, and the trace left by a single loop is looked at. - **AA41.** The setup is common: the **fictional** subscriber records of the measurement network, the **fictional** document vocabulary, five **fictional** tools, twenty-four **fictional** tasks and an **agent simulator** written inside the lesson. No real model endpoint or tool server is called. - **AA42.** The baseline is the **scaffolding-free loop**: the same set, the same budget, the round closed. The plain row with the budget doubled also stays in the table, because the steps the round spends could also have been given to **a single longer run**. - **AA43.** The round **cannot see the correct answer**; it looks only at three observable signs: did the loop stop by itself, did a tool error occur, was an answer produced. - **AA44.** If one of the three is broken, the task counts as **flagged** and the loop runs a second time. The round is **single**, and it can corrupt the answer as well as correct it. - **AA45.** The cost of the round after every run is **one step** and `45 + 18 x len(trace)` tokens; in a run with the round, every tool error adds **40 tokens** for reading the trace. - **AA46.** The second run seed is **20260219**; in this set **the run-to-run difference is one task** (0.0417) and every difference below it **counts as unmeasured**. #### The Three Signs the Round Can Look At The round has no correct answer in hand; if it did, there would be no need for the agent. It has only the trace. ```python # COMMON FICTIONAL SETUP -- the 60 fictional subscribers of the measurement network, # 5 fictional documents, 5 fictional tools, 24 fictional tasks and the agent simulator # written inside the lesson. No real model endpoint, tool server or runtime is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL environment: records of the measurement network REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "previous": previous, "current": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1) def t_read_meter(subscriber=None, **_): a = SUBSCRIBERS.get(subscriber) if a is None: return {"error": "no such subscriber"} return {"previous": a["previous"], "current": a["current"], "type": a["type"], "region": a["region"]} def t_consumption(previous=None, current=None, **_): if previous is None or current is None: return {"error": "missing field"} f = current - previous return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, type="residential", **_): if m3 is None: return {"error": "missing field"} return {"amount": _charge(m3, type)} def t_document(query=None, **_): if not query: return {"error": "missing field"} q = set(tokenize(query)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"} def t_record(subscriber=None, column=None, value=None, **_): if not (subscriber and column and value is not None): return {"error": "missing field"} return {"written": True, "subscriber": subscriber, "column": column, "value": value} TOOLS = { "read_meter": {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False}, "consumption": {"function": t_consumption, "fields": ("previous", "current"), "side_effect": False}, "tariff": {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False}, "search_docs": {"function": t_document, "fields": ("query",), "side_effect": False}, "write_record": {"function": t_record, "fields": ("subscriber", "column", "value"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"error": "no such tool"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) # ---- FICTIONAL tasks: each one is solved by a known tool sequence TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("read_meter", "consumption", "tariff")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("read_meter", "consumption")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("search_docs",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("read_meter", "consumption", "tariff", "write_record")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["type"]) # ---- FICTIONAL agent: at each step it picks a tool or finishes def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["approvals"] = state.get("approvals", 0) + 1 if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["result"] = s["m3"] if choice == "tariff": state["result"] = s["amount"] if choice == "search_docs": state["result"] = s["code"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} # ---- Part added in this lesson: the self-critique round def flagged(s): """The three signs the round can look at: did the loop stop by itself, did a tool error occur, was an answer produced. The round does not have the correct answer.""" return (not s["done"]) or s["errors"] > 0 or s["answer"] is None def review_cost(s): return 45 + 18 * len(s["trace"]) print("task kind done error answer flagged correct") for g in TASKS[:6]: s = agent(g, max_steps=2) print(f"{g['name']:<7}{g['kind']:<12}{str(s['done']):<7}{s['errors']:<6}" f"{str(s['answer'] is not None):<7}{str(flagged(s)):<9}" f"{s['answer'] == correct_answer(g)}") def four_cells(b, seed): c = {"fw": 0, "fr": 0, "sw": 0, "sr": 0} for g in TASKS: s = agent(g, max_steps=b, seed=seed) c[("f" if flagged(s) else "s") + ("r" if s["answer"] == correct_answer(g) else "w")] += 1 return c print("\nbudget seed flagged+wrong flagged+right silent+wrong silent+right") for b in (2, 4, 8): for t in (SEED, SEED + 1): c = four_cells(b, t) print(f"{b:>6}{t:>10}{c['fw']:>14}{c['fr']:>15}{c['sw']:>15}{c['sr']:>14}") print("\nsilent wrong answer (budget 8, seed 20260219):") for g in TASKS: s = agent(g, max_steps=8, seed=SEED + 1) if not flagged(s) and s["answer"] != correct_answer(g): print(" ", g["name"], g["kind"], [a for a, h in s["trace"]]) print(" answer", s["answer"], "-- correct", correct_answer(g)) ``` ``` task kind done error answer flagged correct G01 document False 2 False True False G02 document True 0 True False True G03 consumption False 1 False True False G04 consumption False 0 True True True G05 charge False 0 True True False G06 charge False 0 True True False budget seed flagged+wrong flagged+right silent+wrong silent+right 2 20260218 15 6 0 3 2 20260219 14 7 0 3 4 20260218 3 7 0 14 4 20260219 4 6 0 14 8 20260218 1 6 0 17 8 20260219 0 6 1 17 silent wrong answer (budget 8, seed 20260219): G15 record ['read_meter', 'consumption', 'tariff', 'consumption', 'write_record'] answer 57 -- correct 582 ``` The first table shows why the round is not cheap. At a narrow budget five of the six tasks are flagged; four are really wrong, and one (`G04`) has produced a correct answer. A broken sign is **not proof of a wrong answer, it is a reason to look**. The four-cell table confirms this on the whole set. At narrow budgets **there is no silent wrong answer**; the cost is unnecessary flags, seven against three wrong answers at four steps. At eight steps the table turns around, and the last lines of the block print the task that stayed silent. `G15` is a record task; its sequence is complete, the loop stopped by itself, and no tool error occurred. But after the job was done the agent wandered and called the `consumption` tool again, this call overwrote the accumulated result **with consumption instead of the charge**, and that value was written to the record. The trace looks clean because **every step is valid on its own**; what is broken is the order of the steps. #### What the Round Adds and Multiplies The round is now run as an arrangement. ```python def run(b, seed, critique=False): """critique=False is the plain loop. critique=True: the loop ends, the round looks at the trace, and if flagged the same task runs once more in a second loop.""" CALLS[0] = 0 d = {"right": 0, "steps": 0, "tokens": 0, "flags": 0, "rescued": 0, "broke": 0} for g in TASKS: s = agent(g, max_steps=b, seed=seed, self_critique=critique) d["steps"] += s["steps"] + critique d["tokens"] += s["tokens"] + (review_cost(s) if critique else 0) first = s["answer"] == correct_answer(g) if critique and flagged(s): d["flags"] += 1 s = agent(g, max_steps=b, seed=seed ^ trace_id("tur2|" + g["name"]), self_critique=True) d["steps"] += s["steps"] + 1 d["tokens"] += s["tokens"] + review_cost(s) final = s["answer"] == correct_answer(g) d["right"] += final d["rescued"] += final and not first d["broke"] += first and not final d["calls"] = CALLS[0] return d for t in (SEED, SEED + 1): print(f"\nseed {t}") print("setup success step calls tokens flags rescued broke") for b in (2, 4, 6, 8): for label, critique in (("plain", False), ("round", True)): d = run(b, t, critique) extra = (f"{d['flags']:>8}{d['rescued']:>9}{d['broke']:>6}" if critique else f"{'-':>8}{'-':>9}{'-':>6}") print(f"budget {b:>2}, {label}{d['right']:>6}/24{d['steps']:>7}" f"{d['calls']:>7}{d['tokens']:>10}{extra}") ``` ``` seed 20260218 setup success step calls tokens flags rescued broke budget 2, plain 9/24 48 45 4758 - - - budget 2, round 13/24 135 84 13007 21 4 0 budget 4, plain 21/24 84 64 9402 - - - budget 4, round 23/24 152 94 17052 10 2 0 budget 6, plain 24/24 92 71 10884 - - - budget 6, round 23/24 164 102 19671 9 0 1 budget 8, plain 23/24 97 74 11978 - - - budget 8, round 23/24 155 94 18821 7 1 1 seed 20260219 setup success step calls tokens flags rescued broke budget 2, plain 10/24 48 45 4758 - - - budget 2, round 12/24 135 83 12839 21 4 2 budget 4, plain 20/24 83 67 9366 - - - budget 4, round 22/24 151 96 16822 10 2 0 budget 6, plain 23/24 96 74 11610 - - - budget 6, round 23/24 155 96 18493 7 0 0 budget 8, plain 23/24 98 74 11994 - - - budget 8, round 23/24 152 92 18312 6 0 0 ``` The budget of four steps is the row where the round looks its best: success goes from 21/24 to **23/24**, **two tasks** on both runs. The cost: steps from 84 to **152**, calls from 64 to 94, tokens from 9402 to **17052**. Of the ten flags only two correct an answer. The plain row at eight steps puts this gain in context: **23/24**, 97 steps, 11978 tokens. The plain loop reaches in 97 steps where the round arrived with 152 steps; when the surplus is given to **a longer single run**, the same success is bought more cheaply. At the budget of two steps the gap is sharper: the run with the round gives 13/24 with 135 steps, while the plain run at four steps gives **21/24** with 84 steps. The row at six steps is the clear example of the second claim of the course: the plain loop makes **24/24** there, and when the round is added it goes down to **23/24**. None of the nine flags corrects a wrong answer, and one corrupts a correct one. #### Below the Band Not every difference in this table is a finding. The success of the plain loop is 24/24 and 23/24 on the two seeds at six steps and 23/24 and 23/24 at eight; the measurement band is **one task**. The one-task fall the round costs at six steps and the zero difference at eight are inside this band and **count as unmeasured**: the mechanism can be shown, its effect cannot be measured at this set size. The only gain that stays above the band is the budget of four steps. At eight steps the only concrete number of the round is the cost: seven flags, six of them unnecessary, and zero tasks gained. #### Summary - A stopped agent can look at its own trace but **cannot see the correct answer**; the three signs it can look at are whether it stopped by itself, whether a tool error occurred and whether it produced an answer. - A broken sign is not proof of a wrong answer: at the budget of four steps only three of the ten flags correspond to a wrong answer. - At wide budgets a **wrong answer with a clean trace** appears; a loop that stopped by itself without errors can overwrite the result it accumulated while wandering, and the round cannot see this. - The highest gain of the round is two tasks at the budget of four steps (21/24 to 23/24); steps go from 84 to 152 and tokens from 9402 to 17052. The plain loop at eight steps gives the same 23/24 with 97 steps. - The differences at six and eight steps are inside the band of two runs (one task) and are not counted as findings. #### Next Step In this lesson the one that checks and the one that is checked were **the same loop**, and the round stayed limited to the surface of the trace. One way to take the looking eye **outside the loop** is to split the job and give it to a second agent. The next lesson puts a single agent and two agents side by side on the same task set. What will be measured is not only success; the **tokens of the handed over context** and the **number of fields lost** in the handoff are counted separately, because the cost of a second agent is paid not in steps but in communication. ### Multi-Agent Patterns Source: https://academia.sh/en/courses/ai-agents/multi-agent-patterns Measuring a single agent and two agents side by side on the same task set: when the eight-step budget is divided in two and the job is distributed between a reader and a calculator, overall success stays at 23/24 on both runs, but steps rise from 97 to 140 and the handed-over context carries 363 tokens. The split raises the four-tool record tasks from 2/3 to 3/3 while it lowers the two-tool consumption tasks from 8/8 to 7/8. When the handoff packet is made cheaper, 178 tokens are gained, 68 fields are lost in return and success falls from 23/24 to 18/24; because the lost field is the subscriber type, the charge of commercial subscribers comes out exactly half. When the four-step budget is divided in two, the arrangement falls from 21/24 to 19/24 and from 20/24 to 16/24. The previous lesson left the checking eye inside the loop: the agent looked at its own trace itself and saw nothing when the trace looked clean. The most direct way to take the eye outside is **to split the job**. Splitting is not free. In a single loop the accumulated state sits in a shared dictionary; two agents do not share the same memory, and everything between them has to be **handed over explicitly**. What is handed over is a text, so it is measured in tokens. The split here is by tool set; it is not the planner–executor separation, both agents execute. - **AA47.** The setup is common: the **fictional** subscriber records of the measurement network, the **fictional** document vocabulary, five **fictional** tools, twenty-four **fictional** tasks and an **agent simulator** written inside the lesson. No real model endpoint or tool server is called. - **AA48.** The baseline is the **scaffolding-free loop**: a single agent, the same set, the same total budget. - **AA49.** The arrangement is two roles. The **reader** sees only `read_meter`, `consumption` and `search_docs`, the **calculator** only the `tariff` and `write_record` tools; the loop of a role is the same as the common loop. - **AA50.** The total step budget is **divided in two** and the handoff counts as **one step**. - **AA51.** The **handoff packet** is a text: the fields to be carried are written in the form `name=value`, its cost is the token count of that text, and every field that is not carried is **lost**. The **full handoff** carries eight fields, the **summary handoff** only the fields `subscriber`, `column`, `m3` and `result`. - **AA52.** The second run seed is **20260219**; in this set the run-to-run difference is **one task** (0.0417) and every difference below it **counts as unmeasured**. #### The Field Lost in the Handoff First the two packet forms are put side by side on a single task. `G23` is a record task; its subscriber is commercial. ```python # COMMON FICTIONAL SETUP -- the 60 fictional subscribers of the measurement network, # 5 fictional documents, 5 fictional tools, 24 fictional tasks and the agent simulator # written inside the lesson. No real model endpoint, tool server or runtime is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL environment: records of the measurement network REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "previous": previous, "current": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1) def t_read_meter(subscriber=None, **_): a = SUBSCRIBERS.get(subscriber) if a is None: return {"error": "no such subscriber"} return {"previous": a["previous"], "current": a["current"], "type": a["type"], "region": a["region"]} def t_consumption(previous=None, current=None, **_): if previous is None or current is None: return {"error": "missing field"} f = current - previous return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, type="residential", **_): if m3 is None: return {"error": "missing field"} return {"amount": _charge(m3, type)} def t_document(query=None, **_): if not query: return {"error": "missing field"} q = set(tokenize(query)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"} def t_record(subscriber=None, column=None, value=None, **_): if not (subscriber and column and value is not None): return {"error": "missing field"} return {"written": True, "subscriber": subscriber, "column": column, "value": value} TOOLS = { "read_meter": {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False}, "consumption": {"function": t_consumption, "fields": ("previous", "current"), "side_effect": False}, "tariff": {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False}, "search_docs": {"function": t_document, "fields": ("query",), "side_effect": False}, "write_record": {"function": t_record, "fields": ("subscriber", "column", "value"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"error": "no such tool"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) # ---- FICTIONAL tasks: each one is solved by a known tool sequence TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("read_meter", "consumption", "tariff")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("read_meter", "consumption")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("search_docs",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("read_meter", "consumption", "tariff", "write_record")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["type"]) # ---- FICTIONAL agent: at each step it picks a tool or finishes def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["approvals"] = state.get("approvals", 0) + 1 if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["result"] = s["m3"] if choice == "tariff": state["result"] = s["amount"] if choice == "search_docs": state["result"] = s["code"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} # ---- Part added in this lesson: the two-agent arrangement and the handoff packet ROLE = {"okuyucu": ("read_meter", "consumption", "search_docs"), "hesapci": ("tariff", "write_record")} HANDOFF_FIELDS = ("subscriber", "column", "previous", "current", "type", "region", "m3", "result") SUMMARY_FIELDS = ("subscriber", "column", "m3", "result") # The packet carries Turkish labels and type/region values: their length enters the # token count (abone = subscriber, sutun = column, onceki = previous, simdiki = current, # tip = type, bolge = region, sonuc = result; mesken = residential, isyeri = commercial, # dogu/bati/kuzey/guney/merkez = east/west/north/south/central). PACKET_LABEL = dict(zip(HANDOFF_FIELDS, ("abone", "sutun", "onceki", "simdiki", "tip", "bolge", "m3", "sonuc"))) PACKET_VALUE = dict(zip(TYPE + REGION, ["mesken", "isyeri"] + ["dogu", "bati", "kuzey", "guney", "merkez"])) FIELD_OF = {v: k for k, v in PACKET_LABEL.items()} VALUE_OF = {v: k for k, v in PACKET_VALUE.items()} def role_agent(g, role, state, max_steps, seed): """The same as the common loop; the tool set and the starting state are given from outside.""" r = rng(seed ^ trace_id(role + "|" + g["name"])) names = ROLE[role] seq = [a for a in g["sequence"] if a in names] trace, step, tok, k = [], 0, 0, 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: if r() < 0.88: break choice = names[int(r() * len(names))] else: choice = target if r() < selection(k + 1) else names[int(r() * len(names))] if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append(choice) tok += 30 if "error" in s: continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["result"] = s["m3"] if choice == "tariff": state["result"] = s["amount"] if choice == "search_docs": state["result"] = s["code"] if choice == target: k += 1 return state, step, tok def handoff_packet(state, carried): """The handed-over context is a text; its cost is measured in tokens.""" p = " ".join(f"{PACKET_LABEL[a]}={PACKET_VALUE.get(state[a], state[a])}" for a in carried if a in state) lost = [a for a in HANDOFF_FIELDS if a in state and a not in carried] return p, len(tokenize(p)), lost def unpack_packet(p): d = {} for part in p.split(): a, _, v = part.partition("=") d[FIELD_OF.get(a, a)] = int(v) if v.lstrip("-").isdigit() else VALUE_OF.get(v, v) return d g = TASKS[22] state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} state, step, tok = role_agent(g, "okuyucu", state, 4, SEED) print(g["name"], g["kind"], "-- reader finished, step", step, "tokens", tok) for label, carried in (("full ", HANDOFF_FIELDS), ("summary", SUMMARY_FIELDS)): p, tl, lost = handoff_packet(state, carried) b2, a2, e2 = role_agent(g, "hesapci", unpack_packet(p), 4, SEED) print(f"{label} packet {tl:>2} tokens, lost {len(lost)} {lost}" f" -- answer {b2.get('result')}, correct {correct_answer(g)}") ``` ``` G23 record -- reader finished, step 3 tokens 306 full packet 18 tokens, lost 0 [] -- answer 1132, correct 1132 summary packet 8 tokens, lost 4 ['previous', 'current', 'type', 'region'] -- answer 566, correct 1132 ``` The difference is ten tokens, and its counterpart is a completely wrong answer. Because the summary packet does not carry the `type` field, the `tariff` tool falls back to its own default and calculates the charge at the residential tariff; the result comes out exactly **half**. The tool gives no error, and that value is written to the record. **A lost field shows up not as a tool error but as a wrong but valid result.** #### What the Second Agent Adds and What It Costs Now the whole set is run; the breakdown by kind also stays in the table. Document tasks come out 7/7 in every arrangement and are not in the table. ```python KINDS = ("document", "consumption", "charge", "record") def run(seed, budget, carried=None): """If carried is None it is a single agent. Otherwise the budget is divided in two: the reader finishes its own job, writes the handoff packet, and the calculator starts with the packet.""" CALLS[0] = 0 d = {"right": 0, "steps": 0, "tokens": 0, "handoff": 0, "lost": 0, "kind": {t: [0, 0] for t in KINDS}} for g in TASKS: if carried is None: s = agent(g, max_steps=budget, seed=seed) answer, d["steps"], d["tokens"] = (s["answer"], d["steps"] + s["steps"], d["tokens"] + s["tokens"]) else: state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} state, a1, b1 = role_agent(g, "okuyucu", state, budget // 2, seed) packet, pb, lost = handoff_packet(state, carried) final, a2, b2 = role_agent(g, "hesapci", unpack_packet(packet), budget // 2, seed) d["steps"] += a1 + 1 + a2 d["tokens"] += b1 + pb + b2 d["handoff"] += pb d["lost"] += len(lost) answer = final.get("result", state.get("result")) d["kind"][g["kind"]][1] += 1 d["kind"][g["kind"]][0] += answer == correct_answer(g) d["right"] += answer == correct_answer(g) d["calls"] = CALLS[0] return d for t in (SEED, SEED + 1): print(f"\nseed {t}") print("budget setup consumption charge record total step calls tokens handoff lost") for budget in (4, 8): for name, carried in (("one agent", None), ("two agents, full", HANDOFF_FIELDS), ("two agents, summary", SUMMARY_FIELDS)): d = run(t, budget, carried) p = "".join(f"{d['kind'][x][0]}/{d['kind'][x][1]:<10}" for x in KINDS[1:]) print(f"{budget:>6} {name:<20}{p}{d['right']:>2}/24{d['steps']:>6}" f"{d['calls']:>6}{d['tokens']:>8}{d['handoff']:>8}{d['lost']:>6}") ``` ``` seed 20260218 budget setup consumption charge record total step calls tokens handoff lost 4 one agent 8/8 5/6 1/3 21/24 84 64 9402 0 0 4 two agents, full 6/8 4/6 2/3 19/24 109 63 8162 358 0 4 two agents, summary 6/8 1/6 1/3 15/24 109 63 7984 180 68 8 one agent 8/8 6/6 2/3 23/24 97 74 11978 0 0 8 two agents, full 7/8 6/6 3/3 23/24 140 69 11703 363 0 8 two agents, summary 7/8 2/6 2/3 18/24 140 69 11525 185 68 seed 20260219 budget setup consumption charge record total step calls tokens handoff lost 4 one agent 8/8 4/6 1/3 20/24 83 67 9366 0 0 4 two agents, full 6/8 1/6 2/3 16/24 108 63 8083 361 0 4 two agents, summary 6/8 0/6 1/3 14/24 108 63 7895 173 72 8 one agent 8/8 6/6 2/3 23/24 98 74 11994 0 0 8 two agents, full 8/8 5/6 3/3 23/24 143 75 12358 372 0 8 two agents, summary 8/8 2/6 2/3 19/24 143 75 12170 184 72 ``` At the budget of eight steps overall success is **23/24** in both arrangements and the same on both runs: the second agent adds nothing to the total. The cost is clear — steps from 97 to **140**, the handed-over context **363 tokens**. Calls, on the other hand, fall from 74 to 69, because in the fifteen tasks that are not the calculator's job the agent spends one step and leaves without calling any tool. The communication cost is not a calls item. The breakdown shows what the total hides. The four-tool **record** tasks go from 2/3 to **3/3** on both runs: the split starts the depth from zero for each agent, and a wrong selection comes from two or three tools instead of five. The same mechanism works in reverse on short sequences — the **consumption** tasks go from 8/8 to 7/8 on the first run, and the **charge** tasks from 6/6 to 5/6 on the second. The record subset is three tasks; the direction is the same, its size **counts as unmeasured**. The summary handoff row writes the second cost. Making the packet cheaper brings 363 tokens down to **185**; in return come **68 lost fields** and a success that falls from 23/24 to **18/24**. The loss gathers in the charge tasks: from 6/6 to **2/6**. **A saving measured in tokens is paid for in tasks.** In the rows with the budget of four steps the arrangement stays **below** the single agent: 19/24 and 16/24. The share left to each agent is not enough to finish its sequence. **Splitting a job is an arrangement only if there is enough budget to split.** #### Summary - Two agents do not share a common memory; every field between them is **handed over explicitly** and its cost is measured in tokens. - At an equal total budget the second agent adds **nothing** to success on both runs (23/24 to 23/24); steps go from 97 to 140 and the handed-over context holds 363 tokens. Calls, on the other hand, **fall** from 74 to 69: the communication cost is in the steps item. - The breakdown opens what the total hides: the record tasks go from 2/3 to 3/3 while the consumption and charge tasks lose one task each. - Making the handoff packet cheaper gains 178 tokens, loses 68 fields and lowers success to 18/24; the lost field shows up as **a wrong but valid result**. - When the four-step budget is divided in two, the arrangement falls below the single agent (19/24 and 16/24). #### Next Step In two lessons the checking was first given to the inside of the loop and then to a second agent; in both, the decision stayed in **the scaffolding itself**. In the `G23` task a wrong value was written to a record and nothing stopped it, because `write_record` is the only tool **with a side effect** and was treated the same as the other tools. The next lesson puts an **approval point** in front of that tool: how many records are written with approval open and closed, how many wrong writes the approval stops, and how many times the human is asked for it. ### Keeping the Human in the Loop Source: https://academia.sh/en/courses/ai-agents/keeping-the-human-in-the-loop Measuring the approval point placed in front of the tool with a side effect: with approval off, five records are written in twenty-four fictional tasks and three are wrong, and on the second run eight records are written and six are wrong. When approval is opened only on the call with a side effect, the written records fall to three and the wrong records to one; the cost is seven and eleven questions. When approval is spread to every step, the blocked writes do not change at all, but the questions rise to 74 and 76 and the tokens from 12103 to 14448. Task success stays fixed at 23/24 in all three scopes, so success does not measure the side effect. The number of blocked calls is larger than the number of prevented records, because some of the blocked calls would have failed anyway with a missing field. In the previous two lessons the agent looked at its own trace, then a second agent stepped in; in both, the last word stayed with the scaffolding. Of the five tools of the fictional environment, four **read** and one **writes**: `write_record` leaves a trace that cannot be undone and until now was treated the same as the other four. This lesson puts an **approval point** in front of that tool: how many records are written with approval on and off, how many of them are wrong and how many times the human is asked. Placing the stops according to the **step interval** was measured in the Building Applications with Model APIs course — stopping at every step gets 478 of 480 steps right, stopping once every eight steps gets 355. That procedure is not repeated; here the stop is placed not according to the interval but according to the **side effect**. - **AA53.** The setup is shared: the **fictional** subscriber records of the measurement network, the **fictional** document corpus, five **fictional** tools, twenty-four **fictional** tasks and an **agent simulator** written inside the lesson. No real model endpoint or tool server is called. - **AA54.** The baseline is the **loop without scaffolding**: approval off, a budget of eight steps. - **AA55.** A write is **wrong** in two cases: if the request does not ask for writing (**unintended side effect**) or if the written value is not the correct answer (**wrong value**). - **AA56.** At the approval point the proposed record is shown. Two checks are made by **rule** (does the request ask for writing, is there a value); the third is **recalculation** and is made with a **share of 0.60**. - **AA57.** A rejected call is **not executed at all**; the agent spends its step and continues. Each question also carries a **35 token** state summary. - **AA58.** The **side-effect scope** asks only before `write_record`; the **every-step scope** asks before every step but can still stop only the call with a side effect. - **AA59.** The second run has seed **20260219**; the difference between the two runs is **one task** in this set (0.0417) and every difference below it **counts as unmeasured**. #### Two Writes, Two Results First, two tasks are run in three scopes. `G09` is a document task and does not ask for writing, `G23` is a record task. ```python # COMMON FICTIONAL SETUP -- the 60 fictional subscribers of the measurement network, # 5 fictional documents, 5 fictional tools, 24 fictional tasks and the agent simulator # written inside the lesson. No real model endpoint, tool server or runtime is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL environment: records of the measurement network REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "previous": previous, "current": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1) def t_read_meter(subscriber=None, **_): a = SUBSCRIBERS.get(subscriber) if a is None: return {"error": "no such subscriber"} return {"previous": a["previous"], "current": a["current"], "type": a["type"], "region": a["region"]} def t_consumption(previous=None, current=None, **_): if previous is None or current is None: return {"error": "missing field"} f = current - previous return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, type="residential", **_): if m3 is None: return {"error": "missing field"} return {"amount": _charge(m3, type)} def t_document(query=None, **_): if not query: return {"error": "missing field"} q = set(tokenize(query)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"} def t_record(subscriber=None, column=None, value=None, **_): if not (subscriber and column and value is not None): return {"error": "missing field"} return {"written": True, "subscriber": subscriber, "column": column, "value": value} TOOLS = { "read_meter": {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False}, "consumption": {"function": t_consumption, "fields": ("previous", "current"), "side_effect": False}, "tariff": {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False}, "search_docs": {"function": t_document, "fields": ("query",), "side_effect": False}, "write_record": {"function": t_record, "fields": ("subscriber", "column", "value"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"error": "no such tool"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) # ---- FICTIONAL tasks: each one is solved by a known tool sequence TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("read_meter", "consumption", "tariff")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("read_meter", "consumption")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("search_docs",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("read_meter", "consumption", "tariff", "write_record")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["type"]) # ---- FICTIONAL agent: at each step it picks a tool or finishes def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["approvals"] = state.get("approvals", 0) + 1 if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["result"] = s["m3"] if choice == "tariff": state["result"] = s["amount"] if choice == "search_docs": state["result"] = s["code"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} # ---- Part added in this lesson: approval point in front of the tool with a side effect RECHECK_SHARE = 0.60 def is_wrong(g, value): """A write is wrong: the request does not ask for writing or the value is not the correct answer.""" return "yaz" not in g["request"] or value != correct_answer(g) def give_approval(g, value, r): """At the approval point the proposed record is shown. Two checks are made by rule, the third by recalculation; a human who does not recalculate approves.""" if "yaz" not in g["request"]: return False if value is None: return False if r() < RECHECK_SHARE: return value == correct_answer(g) return True def approved_agent(g, max_steps, seed, scope="off"): """Shared loop. If scope is 'side_effect' the human is asked only before the call with a side effect, if 'every_step' before every step; blocking happens only on the side effect.""" r = rng(seed ^ trace_id(g["name"])) ri = rng(seed ^ trace_id("insan|" + g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, k = [], 0, 0, 0 out = {"writes": 0, "wrong_writes": 0, "blocked": 0, "blocked_wrong": 0, "questions": 0, "written": None} while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = g["sequence"][k] if k < len(g["sequence"]) else None if target is None: if r() < 0.88: break choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] side = TOOLS[choice]["side_effect"] if scope == "every_step" or (scope == "side_effect" and side): out["questions"] += 1 tok += 35 if side and not give_approval(g, state.get("result"), ri): out["blocked"] += 1 out["blocked_wrong"] += is_wrong(g, state.get("result")) trace.append(choice) continue if choice == "write_record": state["value"] = state.get("result") res = call_tool(choice, state) trace.append(choice) tok += 30 if res.get("written"): out["writes"] += 1 out["written"] = res["value"] out["wrong_writes"] += is_wrong(g, state.get("value")) if "error" in res: continue state.update({key: val for key, val in res.items() if key != "error"}) if choice == "consumption": state["result"] = res["m3"] if choice == "tariff": state["result"] = res["amount"] if choice == "search_docs": state["result"] = res["code"] if choice == target: k += 1 out.update(steps=step, tok=tok, answer=state.get("result")) return out print("task kind scope questions writes wrong blocked written value") for i in (8, 22): g = TASKS[i] print(f"-- {g['name']} correct answer {correct_answer(g)}") for name in ("off", "side_effect", "every_step"): s = approved_agent(g, 8, SEED, name) print(f"{g['name']:<6}{g['kind']:<10}{name:<13}{s['questions']:>10}{s['writes']:>8}" f"{s['wrong_writes']:>7}{s['blocked']:>9} {s['written']}") ``` ``` task kind scope questions writes wrong blocked written value -- G09 correct answer SY01 G09 document off 0 1 1 0 SY01 G09 document side_effect 1 0 0 1 None G09 document every_step 2 0 0 1 None -- G23 correct answer 1132 G23 record off 0 1 1 0 TR01 G23 record side_effect 1 1 1 0 TR01 G23 record every_step 8 1 1 0 TR01 ``` The two blocks show two different wrongs. `G09` is a document question; after finding the correct answer the agent wanders and calls `write_record`, leaving **a record nobody asked for**. The written value is right, what is wrong is the writing itself. The approval point stops this by **rule**: if the request does not ask for writing, the proposal is rejected and no calculation is needed. `G23` is the other one. The request asks for writing and there is a value, so both rule checks pass. But before writing, the agent wandered and called `search_docs` and overwrote the accumulated result with a document code: `TR01` is written to the record instead of 1132. The rule check looks at the **presence** of the value, not its **type**; the only way to see it is recalculation, and the human does not recalculate in this run. The record is written in all three scopes. **A rule violation is caught for free; a wrong value is caught only by recalculation.** The cost of the scope is in the questions column: in `G23` asking at every step makes **eight questions**, the side-effect scope asks **once**, and the result is the same. #### What the Approval Stops and What It Costs Now the whole set is run in three scopes. ```python def run(seed, scope, max_steps=8): CALLS[0] = 0 d = {"correct": 0, "steps": 0, "tok": 0, "questions": 0, "writes": 0, "wrong_writes": 0, "blocked": 0} for g in TASKS: s = approved_agent(g, max_steps, seed, scope) d["correct"] += s["answer"] == correct_answer(g) d["steps"] += s["steps"] d["tok"] += s["tok"] for a in ("questions", "writes", "wrong_writes", "blocked"): d[a] += s[a] d["calls"] = CALLS[0] return d for t in (SEED, SEED + 1): print(f"\nseed {t}") print("scope success steps calls tokens questions writes wrong blocked") for scope in ("off", "side_effect", "every_step"): d = run(t, scope) print(f"{scope:<13}{d['correct']:>4}/24{d['steps']:>7}{d['calls']:>7}" f"{d['tok']:>10}{d['questions']:>11}{d['writes']:>8}" f"{d['wrong_writes']:>7}{d['blocked']:>9}") ``` ``` seed 20260218 scope success steps calls tokens questions writes wrong blocked off 23/24 97 74 11978 0 5 3 0 side_effect 23/24 97 70 12103 7 3 1 4 every_step 23/24 97 70 14448 74 3 1 4 seed 20260219 scope success steps calls tokens questions writes wrong blocked off 23/24 98 74 11994 0 8 6 0 side_effect 23/24 100 68 12605 11 3 1 8 every_step 23/24 100 68 14880 76 3 1 8 ``` The first row is the baseline: with approval off the set leaves **five records** and **three are wrong**; on the second run eight records, six wrong. In the same row task success is **23/24**. Side by side, the two numbers make the lesson clear: **task success does not measure the side effect.** While giving the correct answer the agent leaves three wrong records in the environment and success does not see it. When the side-effect scope is opened, the written records fall to **three** and the wrong records to **one**; on the second run also to three and one. The wrong writes blocked are **2** and **5** on the two runs, both above the measurement band. The cost is seven and eleven questions — **3.5** and **2.2** questions per blocked wrong write. Tokens go from 11978 to 12103, calls fall from 74 to 70. The difference between the blocked column and the writes column matters. In the first run the approval point blocks **four** calls but the records it prevents are **two**; the other two calls would already have fallen into the tool's error because there was no value. **A blocked call is not a prevented record.** The third row is an example of the course's second claim. Asking at every step does **not change** the written records, the wrong records and the blocked calls at all — all three columns are exactly the same as in the side-effect scope. The only thing that changes is the cost: questions from 7 to **74**, tokens from 12103 to **14448**. Ten times the questions, zero gain. The only kind of call that can be blocked is the one with a side effect; in the remaining sixty-seven questions the only answer the human can give is approval. #### Summary - A tool with a side effect has to be handled separately: `write_record` leaves a trace that cannot be undone the moment it is called. - With approval off, the set leaves five records and three are wrong (eight and six on the second run) — but in the same run task success is 23/24. **Task success does not measure the side effect.** - When approval is opened on the call with a side effect, the written records fall to three and the wrong records to one; the wrong writes blocked are 2 and 5 on the two runs, at a cost of seven and eleven questions. - A rule violation is caught for free; a wrong value is caught only by **recalculation**. The number of blocked calls is also larger than the number of prevented records (four against two). - Spreading the approval scope to every step does not change the blocked writes at all; it raises the questions from 7 to 74 and the tokens from 12103 to 14448. #### Next Step In seven lessons an agent was built: the loop, step depth, splitting the job, the stopping criterion, a round that looks at its own trace, a second agent and the approval point in front of the side effect. In each lesson the cost was written down and the places where the added step made the job worse were shown with numbers. Under all these measurements there is a silent assumption: **the tool was ready and correct.** `read_meter` always returned the same fields, `tariff` gave a number even when it fell to its default, `write_record` wrote the moment it was asked. How the agent **could select** these tools was never asked: what the name, description and schema said, in what form the result was given back to the model, whether the place where the tool ran had a limit. The next topic looks at **the tool itself** and the first question is the definition: how much the name, description and schema of a tool move the valid call rate. ## Tools and Memory Counting one by one the surfaces the agent touches and the state it carries: the valid call rate and tool errors across the four levels of a tool definition (name only, name and description, schema, schema and example call); the effect of giving a result raw, truncated, or summarized on success and on accumulated tokens; the number of calls that exceed the declared resource limit, with the share of tasks the limit stops and the share that stopping lowers; completed tasks and stopped calls under full, read-only, and allowlisted permission sets; the share of correct answers and the number of steps carrying the wrong source in a mix of three source classes; the tokens carried per step as the loop grows, the steps that fit in the window, and the information that drops when the window fills; the records stored under the every-step, result-only, and selective write policies and the steps gained in later tasks; the tokens that fall as the compaction ratio rises and the number of tasks that need the discarded field; the stale records that fall and the valid-record loss that rises as the expiry threshold tightens. ### Tool Definition Source: https://academia.sh/en/courses/ai-agents/tool-definition Measuring what a tool's name, description, and schema add to the loop: on 24 fictional tasks the definition is given at four levels, and when only the name is given, 113 of 157 calls come back as tool errors, the valid call rate stays at 0.2803, and task success is 0.3333. Adding the description brings the errors down to 56 and raises task success to 0.6667. Adding the schema drops the call count to 74 and the errors to 8, and the valid rate becomes 0.8919, but task success stalls at 0.7917: the schema gives the field names, not which field carries which value, and two swapped indexes return 99948 instead of 52 in a call that passes the schema. The example call zeroes this share, raises task success to 0.9583, and changes the steps, the calls, and the tool errors not at all; its cost is the catalog growing from 97 to 164 tokens, and tokens per correct answer rising from 1126 to 1212. The previous seven lessons built the agent loop: perception, reasoning, the sequence of action and observation, the separation of the planner and the executor, the step budget, the stopping criterion, the self-critique round, the multi-agent arrangement, and the approval point on a tool with a side effect. One thing was silently assumed throughout: **the tool was ready and correct.** This lesson removes that assumption: what does a tool's **definition** add to the loop? The Building Applications with Model APIs course measured the form of a **single call**: with no catalog given the valid call rate was 0.0500, and with names and parameters given together it was 0.6917. That measurement was about parsing a text and is not repeated; the numbers are taken as input. The call here crosses a function boundary, and what is measured is what the definition adds **across the loop**. - **TM1.** The environment, the tool set, the task set, and the agent loop are **fictional**; no real model endpoint, tool server, or runtime is called. 60 fictional subscribers, 5 fictional documents, 5 tools, and 24 tasks each solved by a known tool sequence. Seed **20260218**, second run **20260219**. - **TM2.** The tool definition is given at four levels: name only; name and description; name, description, and schema; schema and example call. Each level includes the one before it. - **TM3.** The definition determines two things: the multiplier on the share of picking the right tool and the **fill share of each declared field**. The name is a weak hint; the description says what the tool does, and the schema says which fields it asks for. - **TM4.** The schema gives the field **names**; it does not say which value a field carries. Two fields of the same type (`onceki` (previous) and `simdiki` (current)) swap with a share of 0.30; **the example call zeroes this share.** - **TM5.** A **tool error** is an error the tool itself returns: a missing field or a nonexistent tool. Fields outside the schema are filtered at the call boundary; a valid call is one that returns no error. - **TM6.** The catalog stays in the prompt at every step; its cost is the token count of the level. - **TM7.** The baseline is the **name only** level; this is the loop without scaffolding. The step budget is eight at all four levels. - **TM8.** **Task success** is the result the loop leaves being equal to the correct answer; it does not depend on whether the agent stopped on its own. #### The Four Levels of Definition The code below builds the environment, the tool set, the task set, and the agent loop, then adds the definition layer on top of the loop. The call boundary still filters fields outside the schema, but whether a declared field is **filled** now depends on the level of the definition. The tool names in the code are the `read_meter`, `consumption`, `tariff`, `search_docs`, and `write_record` tools of the course's earlier lessons; they are counted in tokens here, so they stay Turkish. ```python SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL environment: records of the measurement network # wire words stay Turkish: they are counted in tokens (same rule as the MCP course) # onceki/simdiki/tip/bolge = previous/current/type/region; mesken/isyeri = residential/commercial REGION = ["dogu", "bati", "kuzey", "guney", "merkez"] TYPE = ["mesken", "isyeri"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"bolge": REGION[int(r() * 5)], "tip": TYPE[int(r() * 2)], "onceki": previous, "simdiki": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "isyeri" else 1) def t_read_meter(abone=None, **_): a = SUBSCRIBERS.get(abone) if a is None: return {"hata": "abone yok"} return {"onceki": a["onceki"], "simdiki": a["simdiki"], "tip": a["tip"], "bolge": a["bolge"]} def t_consumption(onceki=None, simdiki=None, **_): if onceki is None or simdiki is None: return {"hata": "eksik alan"} f = simdiki - onceki return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, tip="mesken", **_): if m3 is None: return {"hata": "eksik alan"} return {"tutar": _charge(m3, tip)} def t_document(anahtar=None, **_): if not anahtar: return {"hata": "eksik alan"} q = set(tokenize(anahtar)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"kod": code, "metin": DOCUMENTS[code]} if best > 0 else {"hata": "bulunamadi"} def t_record(abone=None, sutun=None, deger=None, **_): if not (abone and sutun and deger is not None): return {"hata": "eksik alan"} return {"yazildi": True, "abone": abone, "sutun": sutun, "deger": deger} TOOLS = { "sayac_oku": {"function": t_read_meter, "fields": ("abone",), "side_effect": False}, "tuketim": {"function": t_consumption, "fields": ("onceki", "simdiki"), "side_effect": False}, "tarife": {"function": t_tariff, "fields": ("m3", "tip"), "side_effect": False}, "belge_ara": {"function": t_document, "fields": ("anahtar",), "side_effect": False}, "kayit_yaz": {"function": t_record, "fields": ("abone", "sutun", "deger"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) # ---- FICTIONAL tasks: each one is solved by a known tool sequence TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("sayac_oku", "tuketim", "tarife")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("sayac_oku", "tuketim")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("belge_ara",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("sayac_oku", "tuketim", "tarife", "kayit_yaz")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["simdiki"] - a["onceki"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["tip"]) # ---- FICTIONAL agent: at each step it picks a tool or finishes def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth DESCRIPTION = { "sayac_oku": "abonenin onceki ve simdiki sayac endeksini dondurur", "tuketim": "iki endeks arasindaki farki metrekup olarak verir", "tarife": "metrekup ve abone tipinden donem bedelini hesaplar", "belge_ara": "anahtar metne en yakin kurgu belgeyi getirir", "kayit_yaz": "bir abonenin bir sutununa deger yazar ; yan etkilidir"} EXAMPLE = {"sayac_oku": "sayac_oku ( abone = A007 )", "tuketim": "tuketim ( onceki = 41200 , simdiki = 41255 )", "tarife": "tarife ( m3 = 55 , tip = mesken )", "belge_ara": "belge_ara ( anahtar = endeks farki )", "kayit_yaz": "kayit_yaz ( abone = A007 , sutun = bedel , deger = 470 )"} # level : name , selection factor , field fill share , swap share DEFINITION = {1: ("name only", 0.62, 0.45, 0.30), 2: ("name + description", 0.86, 0.72, 0.30), 3: ("name + description + schema", 1.00, 1.00, 0.30), 4: ("schema + example call", 1.00, 1.00, 0.00)} def catalog(level, name): s = name if level >= 2: s += " : " + DESCRIPTION[name] if level >= 3: s += " ( " + " , ".join(TOOLS[name]["fields"]) + " )" if level >= 4: s += " ornek : " + EXAMPLE[name] return s CATALOG_TOKENS = {d: sum(len(tokenize(catalog(d, a))) for a in TOOLS) for d in DEFINITION} def call_defined(name, state, level, rd): """The defined variant of the shared call boundary: whether a declared field is filled depends on the level of the definition.""" CALLS[0] += 1 if name not in TOOLS: return {"hata": "arac yok"} _, _, share, swap = DEFINITION[level] kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state and rd() < share} if {"onceki", "simdiki"} <= set(kw) and rd() < swap: kw["onceki"], kw["simdiki"] = kw["simdiki"], kw["onceki"] return TOOLS[name]["function"](**kw) def agent_defined(g, level=4, max_steps=8, seed=SEED): """The shared agent loop; the only differences are that the call passes through the definition layer and the catalog stays in the prompt at every step.""" r = rng(seed ^ trace_id(g["name"])) rd = rng(seed ^ trace_id(g["name"] + "tanim")) factor = DEFINITION[level][1] state = {"abone": g["subscriber"], "anahtar": g["request"], "sutun": "bedel"} trace, step, tok, errors, foreign = [], 0, 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) + CATALOG_TOKENS[level] target = seq[k] if k < len(seq) else None if target is None: if r() < 0.88: break choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) * factor else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if choice == "kayit_yaz": state["deger"] = state.get("sonuc") s = call_defined(choice, state, level, rd) trace.append((choice, "hata" in s)) tok += 30 if "hata" in s: errors += 1 continue foreign += choice not in seq state.update({kk: vv for kk, vv in s.items() if kk != "hata"}) if choice == "tuketim": state["sonuc"] = s["m3"] if choice == "tarife": state["sonuc"] = s["tutar"] if choice == "belge_ara": state["sonuc"] = s["kod"] if choice == target: k += 1 return {"steps": step, "tokens": tok, "errors": errors, "foreign": foreign, "answer": state.get("sonuc")} for d in DEFINITION: print(f"level {d} {DEFINITION[d][0]:<27} catalog {CATALOG_TOKENS[d]:>3} tokens") print(" level 3 line :", catalog(3, "tuketim")) print(" level 4 suffix : ornek :", EXAMPLE["tuketim"]) g = TASKS[2] for d in (3, 4): s = agent_defined(g, d) print(f" {g['name']} {g['kind']:<11} level {d} answer {s['answer']:>6} correct" f" {correct_answer(g)} tool errors {s['errors']}") ``` ``` level 1 name only catalog 10 tokens level 2 name + description catalog 71 tokens level 3 name + description + schema catalog 97 tokens level 4 schema + example call catalog 164 tokens level 3 line : tuketim : iki endeks arasindaki farki metrekup olarak verir ( onceki , simdiki ) level 4 suffix : ornek : tuketim ( onceki = 41200 , simdiki = 41255 ) G03 consumption level 3 answer 99948 correct 52 tool errors 1 G03 consumption level 4 answer 52 correct 52 tool errors 1 ``` The last two lines are the core of this lesson. The same task, the same seed, the same tool sequence, and the **same number of tool errors**. At the third level the agent called the `tuketim` (consumption) tool with the two fields the schema declares, the call passed the schema, the tool returned no error, and the result came out 99948 instead of 52. The schema declares the names `onceki` and `simdiki`; it does not declare which one is smaller. When the fields swap, the difference goes negative, the meter rollover rule kicks in, and the tool returns a consumption close to a hundred thousand. This call is **valid and wrong.** #### The Numbers of the Four Levels The same 24 fictional tasks run at four definition levels; task success is printed separately for the two seeds. ```python def measure(level, seed=SEED): CALLS[0] = 0 ok = steps = tok = errors = foreign = 0 for g in TASKS: s = agent_defined(g, level, seed=seed) ok += s["answer"] == correct_answer(g) steps += s["steps"] tok += s["tokens"] errors += s["errors"] foreign += s["foreign"] return ok / len(TASKS), CALLS[0], errors, foreign, steps, tok print("definition level calls errors valid task success A task success B" " steps tokens tokens per answer") for d in DEFINITION: a, b = measure(d), measure(d, SEED + 1) print(f"{DEFINITION[d][0]:<27} {a[1]:>5} {a[2]:>6} {(a[1] - a[2]) / a[1]:>6.4f}" f" {a[0]:>14.4f} {b[0]:>14.4f} {a[4]:>5} {a[5]:>6} {a[5] / (a[0] * 24):>17.0f}") a = measure(4) print(f"highest level : {a[3]} of {a[1] - a[2]} valid calls call a tool" f" outside the task's sequence") ``` ``` definition level calls errors valid task success A task success B steps tokens tokens per answer name only 157 113 0.2803 0.3333 0.4167 167 27950 3494 name + description 111 56 0.4955 0.6667 0.5833 129 27489 1718 name + description + schema 74 8 0.8919 0.7917 0.7917 97 21387 1126 schema + example call 74 8 0.8919 0.9583 0.9583 97 27886 1212 highest level : 8 of 66 valid calls call a tool outside the task's sequence ``` The baseline is the first row: when only the name is known, **113** of 157 calls come back as tool errors, the valid call rate is **0.2803**, and task success is **0.3333**. Most of the errors are missing fields. The name does not say what the tool wants; the agent cannot know which key to pass from the state it holds. The cost shows in the steps column: although the budget is eight, the total step count is **167**, more than seven per task. A failed call spends a step and does not advance the sequence. When the description is added, the errors fall to **56**, the calls to **111**, and task success becomes **0.6667**. The description does two jobs: it raises the share of picking the right tool and it hints at what information the fields ask for. When the schema is added, the calls fall to **74**, the errors to **8**, the valid rate becomes **0.8919**, and the steps fall to **97**. The remaining eight errors are unrelated to the schema: a wrongly chosen tool fails its own precondition. #### What the Schema Does Not Say The third and fourth rows carry the same calls, the same errors, and the same steps; their task success separates as **0.7917 and 0.9583**. The 0.1666 between them is what the example call adds, and it arrives without adding a single step to the loop. The run-to-run difference is 0.0834 at the first two levels and zero at the last two; 0.1666 is above twice this band and **counts as measured**. The example call supplies the one piece of information the schema cannot: which value the field carries. The cost is in the tokens column. The catalog grows from 97 tokens to **164 tokens**, and because it stays in the prompt at every step, the total tokens rise from **21387 to 27886**. Tokens per correct answer rise from **1126 to 1212**: while the example call raises task success, at this scale it raises the unit cost as well. A scaffolding whose cost is not written down counts as unmeasured; the cost here is sixty-seven tokens. The last line is the outermost limit of the schema. Even at the highest level, **eight of the 66 valid calls** call a tool that is not in the task's sequence. In the Building Applications with Model APIs course the same residue was fourteen wrong tool names; there the call was a text, here it crossed a function boundary and the tool ran correctly. A definition says what **can be called**; it does not say what **should be called**. #### Summary - The tool definition splits into four levels: name only, name and description, schema, and schema and example call. - When only the name is given, 113 of 157 calls come back as tool errors, the valid rate is 0.2803, and task success is 0.3333; a failed call spends a step and raises the total steps to 167. - The description brings the errors down to 56 and the schema to 8; with the schema the calls fall to 74, the steps to 97, and the valid rate becomes 0.8919. - The schema gives the field names and not which value a field carries: two swapped indexes return 99948 instead of 52 in a call that passes the schema. - The example call raises task success from 0.7917 to 0.9583 and changes the steps, the calls, and the errors not at all; its cost is the catalog growing to 164 tokens and the tokens per correct answer rising from 1126 to 1212. - Even at the highest level, eight of the 66 valid calls call a tool outside the task's sequence; being valid is not being correct. #### Next Step In this lesson the tool was called and the dictionary it returned was written to the state as it was; every field was carried to the next step and stayed in the prompt again at every step. This is one reason the tokens column rises above a thousand tokens per task. Carrying the result as it is is not the only option: it can be truncated, it can be summarized, or only the needed field can be kept. Each lowers the tokens, and each risks dropping something. The next lesson measures **in which form** the result is given back to the model and prints task success side by side with the tokens that accumulate across the loop. ### Executing Tool Calls Source: https://academia.sh/en/courses/ai-agents/executing-tool-calls Measuring in which form the result a tool returns is given back to the model: on 24 fictional tasks the raw observation gives a task success of 0.9583 with 770 observation tokens and 10103 total tokens. Truncating the observation to seven words drops the tokens to 9160, but because it drops the subscriber type, four commercial invoices come out at exactly half and task success falls to 0.7917. Truncating to four words breaks the chain: tool errors jump from 8 to 116, steps from 97 to 156, total tokens to 15945, and task success stays at 0.2917; making it cheaper makes it more expensive. The summary that declares the fields to carry keeps the task success of the raw observation in both runs while dropping the observation tokens to 401 and the tokens per correct answer from 439 to 401. When the subscriber type is removed from the same summary, task success falls to the truncation level, 0.7917: a summary is only as good as its declaration. In the previous lesson the tool definition was split into four levels, and at the highest level the valid call rate was 0.8919 and task success 0.9583. That measurement never questioned the **return** of the call: whatever the tool returned was written to the state with all its fields and carried to the next step. In a loop this is a shaky assumption. A tool result is an **observation** given back to the model, and that observation sits in the prompt again at every step. The full text a document tool returns, the four fields a reading tool returns, the confirmation of a write tool: all of them accumulate as the loop grows. The question of this lesson is this: in what form should the observation be given back, what happens to task success, and how many tokens accumulate across the loop. - **TM9.** The environment, the tool set, the task set, and the agent loop are **fictional**; no real model endpoint, tool server, or runtime is called. 60 fictional subscribers, 5 fictional documents, 5 tools, and 24 tasks each solved by a known tool sequence. Seed **20260218**, second run **20260219**. - **TM10.** A tool returns a dictionary; the **observation** is that dictionary turned into text. The agent updates its state not from the dictionary but **by reading back from the observation text**. - **TM11.** There are three observation forms. The **raw** observation carries all the fields. The **truncated** observation cuts the text at a given number of words; a field whose value is left incomplete at the cut cannot be read back. The **summarized** observation carries the fields **declared** per tool. - **TM12.** Truncation is measured at two thresholds: seven words and four words. - **TM13.** The accumulated observation is sent again in the prompt at every step. The shared loop counts this accumulation with a constant coefficient per step; this lesson puts **the token count of the observation actually carried** in place of that coefficient. Tokens are counted with the fictional tokenizer. - **TM14.** The observation of a call that returns an error also accumulates; an error text is also an observation. - **TM15.** The baseline is the **raw observation**; this is the loop without scaffolding. Truncation and summarizing are the scaffolding added. The step budget is eight in all four forms. - **TM16.** Task success is the result the loop leaves being equal to the correct answer. #### The Three Forms of Observation The code below builds the shared environment, the tool set, and the task set, then adds the observation layer: a writer that turns the dictionary into text, a parser that reads the fields back from the text, and the three forms. The tool and field names in the code are the `read_meter`, `consumption`, `tariff`, `search_docs`, and `write_record` tools of the previous lessons; they are counted in tokens here, so they stay Turkish. ```python SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL environment: records of the measurement network # wire words stay Turkish: they are counted in tokens (same rule as the MCP course) # onceki/simdiki/tip/bolge = previous/current/type/region; mesken/isyeri = residential/commercial REGION = ["dogu", "bati", "kuzey", "guney", "merkez"] TYPE = ["mesken", "isyeri"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"bolge": REGION[int(r() * 5)], "tip": TYPE[int(r() * 2)], "onceki": previous, "simdiki": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "isyeri" else 1) def t_read_meter(abone=None, **_): a = SUBSCRIBERS.get(abone) if a is None: return {"hata": "abone yok"} return {"onceki": a["onceki"], "simdiki": a["simdiki"], "tip": a["tip"], "bolge": a["bolge"]} def t_consumption(onceki=None, simdiki=None, **_): if onceki is None or simdiki is None: return {"hata": "eksik alan"} f = simdiki - onceki return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, tip="mesken", **_): if m3 is None: return {"hata": "eksik alan"} return {"tutar": _charge(m3, tip)} def t_document(anahtar=None, **_): if not anahtar: return {"hata": "eksik alan"} q = set(tokenize(anahtar)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"kod": code, "metin": DOCUMENTS[code]} if best > 0 else {"hata": "bulunamadi"} def t_record(abone=None, sutun=None, deger=None, **_): if not (abone and sutun and deger is not None): return {"hata": "eksik alan"} return {"yazildi": True, "abone": abone, "sutun": sutun, "deger": deger} TOOLS = { "sayac_oku": {"function": t_read_meter, "fields": ("abone",), "side_effect": False}, "tuketim": {"function": t_consumption, "fields": ("onceki", "simdiki"), "side_effect": False}, "tarife": {"function": t_tariff, "fields": ("m3", "tip"), "side_effect": False}, "belge_ara": {"function": t_document, "fields": ("anahtar",), "side_effect": False}, "kayit_yaz": {"function": t_record, "fields": ("abone", "sutun", "deger"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"hata": "arac yok"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) # ---- FICTIONAL tasks: each one is solved by a known tool sequence TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("sayac_oku", "tuketim", "tarife")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("sayac_oku", "tuketim")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("belge_ara",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("sayac_oku", "tuketim", "tarife", "kayit_yaz")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["simdiki"] - a["onceki"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["tip"]) # ---- FICTIONAL agent: at each step it picks a tool or finishes def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth # The dictionary a tool returns is turned into text before it is given back to the model. SUMMARY_FIELDS = {"sayac_oku": ("onceki", "simdiki", "tip"), "tuketim": ("m3",), "tarife": ("tutar",), "belge_ara": ("kod",), "kayit_yaz": ("yazildi",)} FORMAT = {"raw": None, "truncated 7": 7, "truncated 4": 4, "summarized": None} def to_text(s): return " , ".join(f"{k} = {v}" for k, v in s.items()) def parse(text): """Reads the fields back from the observation text; a field whose value was cut off is dropped.""" d = {} for p in text.split(" , "): a, _, v = p.partition(" = ") v = v.strip(" ,") if v: d[a.strip(" ,")] = int(v) if v.lstrip("-").isdigit() else v return d def observe(name, result, form): """Raw , truncated , or summarized observation text.""" if form == "summarized": result = {k: v for k, v in result.items() if k in SUMMARY_FIELDS.get(name, ())} text = to_text(result) if FORMAT[form]: text = " ".join(text.split()[:FORMAT[form]]) return text def agent_observation(g, form="raw", max_steps=8, seed=SEED): """The shared agent loop; the only differences are that the observation is turned into text and read back, and that the accumulated observation tokens are sent again at every step.""" r = rng(seed ^ trace_id(g["name"])) state = {"abone": g["subscriber"], "anahtar": g["request"], "sutun": "bedel"} trace, step, tok, errors, accum = [], 0, 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + accum # the accumulated observation is sent again at every step target = seq[k] if k < len(seq) else None if target is None: if r() < 0.88: break choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if choice == "kayit_yaz": state["deger"] = state.get("sonuc") s = call_tool(choice, state) trace.append((choice, "hata" in s)) tok += 30 text = observe(choice, s, form) accum += len(tokenize(text)) if "hata" in s: errors += 1 continue d = parse(text) state.update(d) if choice == "tuketim" and "m3" in d: state["sonuc"] = d["m3"] if choice == "tarife" and "tutar" in d: state["sonuc"] = d["tutar"] if choice == "belge_ara" and "kod" in d: state["sonuc"] = d["kod"] if choice == target: k += 1 return {"steps": step, "tokens": tok, "errors": errors, "observation": accum, "answer": state.get("sonuc")} for name in ("sayac_oku", "belge_ara"): sample = TOOLS[name]["function"](**{"abone": "A007", "anahtar": "endeks farki"}) for form in ("raw", "truncated 7", "summarized"): m = observe(name, sample, form) print(f"{name:<10} {form:<11} {len(tokenize(m)):>2} tokens {m}") print(f"{name:<10} truncated 7 fields read back :", " , ".join(parse(observe(name, sample, "truncated 7")))) ``` ``` sayac_oku raw 19 tokens onceki = 78189 , simdiki = 78241 , tip = mesken , bolge = merkez sayac_oku truncated 7 9 tokens onceki = 78189 , simdiki = 78241 sayac_oku summarized 14 tokens onceki = 78189 , simdiki = 78241 , tip = mesken sayac_oku truncated 7 fields read back : onceki , simdiki belge_ara raw 19 tokens kod = SY01 , metin = endeks farki tuketimi verir sayac donusu 100000 eklenir belge_ara truncated 7 8 tokens kod = SY01 , metin = endeks belge_ara summarized 3 tokens kod = SY01 belge_ara truncated 7 fields read back : kod , metin ``` Both tools carry the same raw length, nineteen tokens, and the two forms cut them in very different places. Truncation looks at **position**: in the `sayac_oku` (read meter) observation the last two fields, `tip` (type) and `bolge` (region), drop because they remain after the seventh word. The summary looks at **declaration**: in the same tool the type stays because it is declared, and the region goes because it is not. In the document tool the order is reversed; there truncation keeps the code because it comes first, while the summary throws away the whole text and drops the nineteen tokens to three. #### The Numbers of the Four Forms The same 24 fictional tasks run in four forms. Observation tokens are the total length of the observation texts produced across the loop; total tokens also include their **being sent again**. ```python def measure(form, seed=SEED): CALLS[0] = 0 ok = steps = tok = errors = obs = 0 for g in TASKS: s = agent_observation(g, form, seed=seed) ok += s["answer"] == correct_answer(g) steps += s["steps"] tok += s["tokens"] errors += s["errors"] obs += s["observation"] return ok / len(TASKS), CALLS[0], errors, steps, obs, tok print("observation form calls errors steps observation tokens total tokens" " task success A task success B tokens per answer") for form in FORMAT: a, c = measure(form), measure(form, SEED + 1) print(f"{form:<16} {a[1]:>6} {a[2]:>6} {a[3]:>5} {a[4]:>18} {a[5]:>12}" f" {a[0]:>14.4f} {c[0]:>14.4f} {a[5] / (a[0] * 24):>17.0f}") SUMMARY_FIELDS["sayac_oku"] = ("onceki", "simdiki") # type dropped from the declaration a = measure("summarized") print(f"narrow summary ( type not declared ) : task success {a[0]:.4f} , observation tokens" f" {a[4]} , tool errors {a[2]}") ``` ``` observation form calls errors steps observation tokens total tokens task success A task success B tokens per answer raw 74 8 97 770 10103 0.9583 0.9583 439 truncated 7 74 8 97 428 9160 0.7917 0.7500 482 truncated 4 149 116 156 616 15945 0.2917 0.2917 2278 summarized 74 8 97 401 9229 0.9583 0.9583 401 narrow summary ( type not declared ) : task success 0.7917 , observation tokens 311 , tool errors 8 ``` The baseline is the first row: the raw observation produces 770 observation tokens, spends 10103 total tokens, and holds task success at **0.9583**. The second row puts side by side what truncation adds and what it drops. Observation tokens fall to **428**, total tokens to **9160**, but task success becomes **0.7917**. Calls, steps, and tool errors do not change at all; the four tasks that drop are all commercial subscribers, and in all four the invoice comes out exactly **half**. When the type drops, the tariff tool uses the declared default, that is residential. Truncation produced no error; it produced a silent wrong answer. The third row is a clear example of the course's second claim. Squeezing the truncation to four words cuts the second index in the `sayac_oku` observation as well; the `tuketim` (consumption) tool returns with a missing field, the chain breaks, and the agent retries the same step. Tool errors rise **from 8 to 116**, calls from 74 to **149**, steps from 97 to **156**. The result: observation tokens are still lower than in the raw form (616), but **total tokens jump to 15945**, one and a half times the raw form. Making the observation cheaper made the loop more expensive. #### Declaration and Position The last row is the number of the summary and it stands alone in the table: task success is **0.9583** in both runs, the same as the raw observation; observation tokens are **401**, less than half of the raw; tokens per correct answer fall **from 439 to 401**. The summary is both the cheapest and the most accurate form, because it reads what to throw away not from position but from **declaration**. This advantage is not free, and the last output line writes its cost. When only the `tip` field is dropped from the summary's declaration, task success falls to **0.7917**, to where truncation dropped it. A summary is only as good as its declaration; whoever knows which field will be needed at the next step must declare it, and if one does not know this, a summary is no safer than truncation. There is one row in the table that looks reversed: the summary's observation tokens are lower than those of truncated 7 (401 against 428) but its total tokens are higher (9229 against 9160). The reason is the arithmetic of accumulation. An observation is sent again at every step that comes after it; therefore **an early observation is costlier than a late one**. The summary keeps the `sayac_oku` observation at 14 tokens (truncation at 9), and that observation is in the **first** step of most tasks. In contrast, the summary's big gain is in `belge_ara` (search docs; from 19 to 3), and that observation is the **last** step of single-step tasks, so it is never sent again. #### Summary - A tool result is not written to the state directly: it is turned into an observation text, read back, and sent again in the prompt at every step. - The raw observation gives 770 observation tokens, 10103 total tokens, and a task success of 0.9583; this is the baseline. - Truncating to seven words drops the tokens to 9160 but, because it drops the subscriber type, halves four commercial invoices and pulls task success to 0.7917; the tool errors do not rise at all, the wrong answer is silent. - Truncating to four words breaks the chain: tool errors rise to 116, steps to 156, total tokens to 15945, and task success falls to 0.2917; making the observation cheaper makes the loop more expensive. - The summary with a declaration keeps the task success of the raw observation while dropping the observation tokens to 401 and the tokens per correct answer to 401; when the type is removed from the same summary, task success falls to 0.7917. - In an accumulating context an early observation is costlier than a late one: an observation is sent again at every step after it. #### Next Step In these two lessons the tools were defined, called, and their returns formatted. All of them had one thing in common: each tool did a **fixed job** and returned. A reading is a fixed number of fields, a computation a fixed number of operations, a search a fixed text. That is why how much a call would cost was never asked. In a tool that runs a program the agent **wrote itself**, this assumption lifts: how many steps it will take, how many cells it will occupy, and how many outputs it will produce are not written in the definition and are seen only while it runs. The next lesson builds such a tool inside a sandbox with resource limits, counts the calls that exceed the declared limit, and writes separately how many tasks stopping it drops. ### Code Execution Tools Source: https://academia.sh/en/courses/ai-agents/code-execution-tools Measuring, through its resource limits, a tool that runs the program an agent wrote inside a sandbox: three ways of writing a program that computes the same amount spend 15, 172, and 19 steps, hold 2 to 5 cells, and produce 1 to 5 outputs. At the loose limit no call exceeds the declared limit, but three tasks pass the 96-step task budget while the per-call limit stops nothing. The medium limit stops 24 of 28 code calls, and the classes it stops are 20 for steps and 4 for cells; at the tight limit the same 24 calls stop this time on 20 step and 4 output classes. Stopping does not drop a task directly; it drops it by consuming the step budget: on code tasks task success falls from 0.7778 to 0.3333, and in the second run from 1.0000 to 0.5556. The tight limit gives the same task success as the medium limit while cutting wasted steps from 1356 to 552. Running code is entirely on the defensive side. In the previous two lessons every tool did a **fixed job** and returned, so what a call would cost was never asked. In a tool that runs a program the agent **wrote itself**, this assumption lifts: two programs that give the same result can spend a hundred times different resources, and that difference is not written in the definition. The lesson builds such a tool inside a **sandbox** and is not an attack lesson: no escape, bypass, or privilege escalation routine is written. **Running code is entirely on the defensive side** — it counts, limits, and stops. Restricting privileges was measured in the Linux and System Administration and Cybersecurity curricula as the **principle of least privilege** and is not repeated; the only things counted are what the limit stops and what stopping costs. - **TM17.** The environment, the tool set, the task set, the code execution tool, and the agent loop are **fictional**; no real runtime, execution service, or model endpoint is called. The tariff computation stops being a fixed tool; `tariff` drops out and `run_code` takes its place, so the tool set stays at five. Seed **20260218**, second run **20260219**. - **TM18.** The sandbox is a small register machine and carries **three declared resource limits**: steps, cells, outputs. When a limit is exceeded the run stops and **the exceeded class is reported**. - **TM19.** The program the agent writes takes one of three forms, **the form is drawn once per task**, and all three forms compute the same amount. - **TM20.** The **actual consumption** of a call is measured with a dry run: the program is run once more with a nearly unlimited setting, and the result enters only the count. - **TM21.** The **task budget** is 96 execution steps and looks at the total of all the code calls of the task; the sandbox does not see this, because its limit is **per call**. - **TM22.** The baseline is **the loose limit**, the step budget is eight, and a code task is one whose sequence contains `run_code`: nine of the 24 tasks. #### The Sandbox and the Three Ways of Writing The code below builds the shared environment and the task set, then adds the register machine, the declared limits, and the compiler. ```python SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL environment: records of the measurement network REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "previous": previous, "current": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1) def t_read_meter(subscriber=None, **_): a = SUBSCRIBERS.get(subscriber) if a is None: return {"error": "no such subscriber"} return {"previous": a["previous"], "current": a["current"], "type": a["type"], "region": a["region"]} def t_consumption(previous=None, current=None, **_): if previous is None or current is None: return {"error": "missing field"} f = current - previous return {"m3": f + 100000 if f < 0 else f} def t_document(query=None, **_): if not query: return {"error": "missing field"} q = set(tokenize(query)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"} def t_record(subscriber=None, column=None, value=None, **_): if not (subscriber and column and value is not None): return {"error": "missing field"} return {"written": True, "subscriber": subscriber, "column": column, "value": value} TOOLS = { "read_meter": {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False}, "consumption": {"function": t_consumption, "fields": ("previous", "current"), "side_effect": False}, "search_docs": {"function": t_document, "fields": ("query",), "side_effect": False}, "write_record": {"function": t_record, "fields": ("subscriber", "column", "value"), "side_effect": True}, } def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"error": "no such tool"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) # ---- FICTIONAL tasks: each one is solved by a known tool sequence TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("read_meter", "consumption", "tariff")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("read_meter", "consumption")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("search_docs",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("read_meter", "consumption", "tariff", "write_record")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["type"]) # ---- FICTIONAL agent: at each step it picks a tool or finishes def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth # ---- FICTIONAL code execution tool: register machine and declared resource limits. # Running code is on the defensive side: it counts , limits , stops. LIMITS = {"loose": {"steps": 512, "cells": 16, "outputs": 16}, "medium": {"steps": 64, "cells": 4, "outputs": 8}, "tight": {"steps": 24, "cells": 4, "outputs": 2}} TASK_BUDGET = 96 # total execution steps per task def val(x, h): return h[x] if isinstance(x, str) else x def execute(program, limit): """When a limit is exceeded the run stops and the exceeded class is reported.""" cells, outputs, ip, step = {}, [], 0, 0 while ip < len(program): step += 1 if step > limit["steps"]: return {"error": "limit exceeded", "class": "step", "spent": step} k = program[ip] if k[0] == "set": cells[k[1]] = val(k[2], cells) elif k[0] == "add": cells[k[1]] += val(k[2], cells) elif k[0] == "mul": cells[k[1]] *= val(k[2], cells) elif k[0] == "dec": cells[k[1]] -= 1 elif k[0] == "write": outputs.append(cells[k[1]]) elif k[0] == "branch" and cells[k[1]] > 0: ip = k[2] continue if len(cells) > limit["cells"]: return {"error": "limit exceeded", "class": "cell", "spent": step} if len(outputs) > limit["outputs"]: return {"error": "limit exceeded", "class": "output", "spent": step} ip += 1 return {"amount": outputs[-1] if outputs else None, "spent": step, "cells": len(cells), "outputs": len(outputs)} def compile_program(style, m3, type_): """The FICTIONAL program the agent writes; all three compute the same amount.""" parts, remaining, lower = [], m3, 0 for upper, unit in TIERS: # split into tariff tiers share = min(remaining, upper - lower) if share <= 0: break parts.append((share, unit)) remaining, lower = remaining - share, upper if remaining > 0: parts.append((remaining, ABOVE)) p = [("set", "t", 0)] for i, (count, unit) in enumerate(parts): if style == "short": p += [("set", "x", count), ("mul", "x", unit), ("add", "t", "x")] elif style == "unit": p += [("set", "s", count), ("add", "t", unit), ("dec", "s"), ("branch", "s", len(p) + 1)] else: # explicit : a cell and an output per tier p += [("set", f"d{i}", count), ("mul", f"d{i}", unit), ("add", "t", f"d{i}"), ("write", f"d{i}")] if type_ == "commercial": p.append(("mul", "t", COMMERCIAL_FACTOR)) return p + [("write", "t")] SETTING = ["loose"] # the limit the sandbox declares def t_code(m3=None, type="residential", program=None, **_): if m3 is None or program is None: return {"error": "missing field"} return execute(program, LIMITS[SETTING[0]]) # The tariff computation is not a fixed tool , it is a program the agent writes; the tool count stays five. TOOLS["run_code"] = {"function": t_code, "fields": ("m3", "type", "program"), "side_effect": False} TOOL_NAMES = list(TOOLS) for g in TASKS: g["sequence"] = tuple("run_code" if a == "tariff" else a for a in g["sequence"]) STYLE, STYLE_SHARE = ("short", "unit", "explicit"), [0.40, 0.30, 0.30] for style in STYLE: p = compile_program(style, 55, "commercial") s = execute(p, LIMITS["loose"]) d = " ".join(f"{a} -> {execute(p, LIMITS[a]).get('class', 'ok')}" for a in ("medium", "tight")) print(f"{style:<8} commands {len(p):>2} steps {s['spent']:>3} cells {s['cells']}" f" outputs {s['outputs']} amount {s['amount']} | {d}") ``` ``` short commands 15 steps 15 cells 2 outputs 1 amount 1100 | medium -> ok tight -> ok unit commands 19 steps 172 cells 2 outputs 1 amount 1100 | medium -> step tight -> step explicit commands 19 steps 19 cells 5 outputs 5 amount 1100 | medium -> cell tight -> output ``` The three rows compute the same amount, 1100, with three different resource profiles. The command counts are nearly equal — 15, 19, 19 — and **no limit can be read from that number**. The form that adds unit by unit spends **172 steps**, because four of its commands are a backward branch; the explicit form finishes in nineteen steps but opens **five cells** and produces **five outputs**. A resource limit is therefore not singular: the step limit stops the second form, and the third is stopped by the cell limit at the medium setting and by the output limit at the tight setting. #### What the Limit Stops and What It Cannot The same 24 tasks run at three limit settings and with two seeds; the calls that exceed the limit are found with a **dry run**, while the active limit really stops. ```python def agent_code(g, max_steps=8, seed=SEED): """The shared agent loop; the only difference is that the program the agent wrote runs in the sandbox instead of the tariff, and every run is recorded.""" r = rng(seed ^ trace_id(g["name"])) style = STYLE[pick(r(), STYLE_SHARE)] # a single way of writing per task state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} code_trace, step, seq, k = [], 0, g["sequence"], 0 while step < max_steps: step += 1 target = seq[k] if k < len(seq) else None if target is None: if r() < 0.88: break choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if choice == "write_record": state["value"] = state.get("outcome") prog = None if choice == "run_code" and "m3" in state: prog = state["program"] = compile_program(style, state["m3"], state.get("type", "residential")) s = call_tool(choice, state) if prog is not None: code_trace.append((prog, s)) if "error" in s: continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["outcome"] = s["m3"] if choice == "run_code": state["outcome"] = s["amount"] if choice == "search_docs": state["outcome"] = s["code"] if choice == target: k += 1 return {"steps": step, "code_trace": code_trace, "answer": state.get("outcome")} def measure(setting, seed=SEED): """Limit check: the dry run gives the actual consumption , the active limit stops.""" SETTING[0], lim, classes = setting, LIMITS[setting], {} ok = code = exceeding = stopped = wasted = over_budget = ok_code = n_code = 0 for g in TASKS: s = agent_code(g, seed=seed) d = s["answer"] == correct_answer(g) ok += d if "run_code" in g["sequence"]: n_code, ok_code = n_code + 1, ok_code + d total = 0 for prog, outcome in s["code_trace"]: dry = execute(prog, LIMITS["loose"]) # for counting only code += 1 exceeding += (dry["spent"] > lim["steps"] or dry["cells"] > lim["cells"] or dry["outputs"] > lim["outputs"]) total += outcome["spent"] if "error" in outcome: stopped += 1 wasted += outcome["spent"] classes[outcome["class"]] = classes.get(outcome["class"], 0) + 1 over_budget += total > TASK_BUDGET return code, exceeding, stopped, wasted, over_budget, ok / len(TASKS), ok_code / n_code, classes print("limit code calls exceeding stopped wasted steps over budget success code tasks") for setting in LIMITS: for seed, e in ((SEED, "A"), (SEED + 1, "B")): a = measure(setting, seed) print(f"{setting:<7}{e} {a[0]:>10} {a[1]:>9} {a[2]:>7} {a[3]:>12} {a[4]:>11}" f" {a[5]:>7.4f} {a[6]:>10.4f}") for setting in ("medium", "tight"): print(f" {setting:<6} stopping classes :", measure(setting)[7]) ``` ``` limit code calls exceeding stopped wasted steps over budget success code tasks loose A 9 0 0 0 3 0.8750 0.7778 loose B 10 0 0 0 1 1.0000 1.0000 medium A 28 24 24 1356 4 0.7083 0.3333 medium B 22 17 17 646 2 0.8333 0.5556 tight A 28 24 24 552 4 0.7083 0.3333 tight B 22 17 17 317 1 0.8333 0.5556 medium stopping classes : {'step': 20, 'cell': 4} tight stopping classes : {'step': 20, 'output': 4} ``` The second and third columns are **equal** in every row: the number of calls that exceed the declared limit and the number of calls stopped are the same. A limit placed per call catches every per-call violation; this is not an achievement but the requirement of the definition. What is worth counting is the place the limit **does not see**. What it does not see is in the fifth column. At the loose setting no call is stopped, yet **three tasks** pass the 96-step budget; a limit that allows 512 steps per call does not count how many times a 172-step program was run. At the medium and tight settings the stopped calls are repeated, so the number of tasks that exceed the budget rises to **four**. Both say the same thing: **a per-call limit does not see the task total**. The cost of stopping is in the last column. At the loose setting, task success on code tasks is **0.7778** and **1.0000**; at the medium setting it is **0.3333** and **0.5556**. The drop is close to 0.4445 in both runs and is twice the run-to-run difference at the loose setting (0.2222), so it is **measured**. The mechanism is indirect: a stopped call does not drop the task, it **consumes the step budget**. Because the way of writing is fixed per task, the same program is written again and the budget runs out; this is how code calls rise from 9 to 28. The only column that separates the medium and tight rows is wasted steps: **1356 against 552**. With the stopped calls and the task success the same, the tight limit recovers three fifths of the discarded work; the gain is not in accuracy but in resources. #### Summary - Three programs that give the same amount spend 15, 172, and 19 steps, hold 2 to 5 cells, and produce 1 to 5 outputs; a resource limit is therefore plural, not singular. - In every row the number of exceeding calls equals the number of stopped calls: a per-call limit catches every per-call violation. The place it does not see is the task total: at the loose setting three tasks pass the 96-step budget with no call stopped, and at the limited settings four tasks do. - Stopping does not drop a task directly, it drops it by consuming the step budget: on code tasks task success falls from 0.7778 and 1.0000 to 0.3333 and 0.5556, and code calls rise from 9 to 28. The tight limit gives the same task success as the medium limit and cuts wasted steps from 1356 to 552. #### Next Step What was limited here was **consumption**: how many steps a call would take, how many cells it would hold, how many outputs it would produce. A resource limit does not touch a fourth question at all: what the tool **can reach**. The program in the sandbox had no outside; the other surfaces the agent touches are not like that. A file tree or a shell can do an irreversible job without lengthening the execution at all. The next lesson counts the **permission surface a tool declares**. ### File System and Terminal Access Source: https://academia.sh/en/courses/ai-agents/file-system-and-terminal-access Counting permission restriction on the agent's file and command surface: twenty-four fictional surface tasks are run under three permission sets, and completed tasks and stopped calls are printed side by side. Under full permission 0.9583 of the tasks are completed and the gate stops no call; the read-only permission stops 71 calls but drops completed tasks to 0.5000; the allowlisted set stops 20 calls and stays at 0.8333, and all of its stops are gathered in a single task kind. Under read-only the correct answer is 0.6250 while completed tasks are 0.5000: in three tasks the answer was produced correctly but the write never happened. A stopped call also consumes a step; steps rise from 78 to 129. Although the totals stay the same in the two runs, two tasks swap places at the task level, so the measurement band is 0.0833 and the 0.1250 difference between full permission and the allowlist is above the band. The previous lesson tied the code execution tool to a resource limit and counted how many calls the limit stopped. What stopped there was **quantity**: memory, time, output length. The surface the agent touches does not end there: the same loop reads files, writes files, and runs operating system commands. In these three actions what stops is not quantity but **permission**. The lesson gathers that permission in a single concept: the **permission surface**, which is the capability a tool declares together with the path or command it touches. The principle of least privilege was established in the Introduction to Cybersecurity course, and file permissions and ownership were measured in the Introduction to Linux course; neither routine is repeated. The only things counted are these: under three permission sets, how many tasks the agent finishes, how many calls the gate stops, and who is billed for stopping. - **TM31.** The code is a **simulator**; no real file system or runtime is called. The environment, the tools, and the tasks are **fictional**, seed **20260218**. - **TM32.** The agent loop is the shared definition of the course: the share of picking the right tool falls with depth, and when the job is done the agent stops with a share of 0.88. - **TM33. The permission surface is the field the tool declares:** each tool declares a **capability** (read, write, execute); the path or command it touches is read from inside the call. The fictional tree carries five paths, and the fictional terminal surface carries four commands. - **TM34. The gate stands in front of the executor.** A call that does not pass never reaches the tool; it returns to the loop as a tool error and consumes a step. The three measured sets are **full**, **read-only**, and **allowlisted**. - **TM35. Two conditions are required to call a task completed:** the answer must be correct and the step with a side effect in the sequence must have run. Looking only at the answer counts a task that did not write as finished. - **TM36. The baseline is the full permission set** — the loop in which the gate stops nothing. The cost is **stopped calls, steps, and dropped tasks**. #### The Shared Loop and the Declaration of the Surface The environment, the five basic tools, and the agent loop are the same throughout the course. ```python SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL environment: records of the measurement network REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "previous": previous, "current": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1) def t_read_meter(subscriber=None, **_): a = SUBSCRIBERS.get(subscriber) if a is None: return {"error": "no such subscriber"} return {"previous": a["previous"], "current": a["current"], "type": a["type"], "region": a["region"]} def t_consumption(previous=None, current=None, **_): if previous is None or current is None: return {"error": "missing field"} f = current - previous return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, type="residential", **_): if m3 is None: return {"error": "missing field"} return {"amount": _charge(m3, type)} def t_document(query=None, **_): if not query: return {"error": "missing field"} q = set(tokenize(query)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"} def t_record(subscriber=None, column=None, value=None, **_): if not (subscriber and column and value is not None): return {"error": "missing field"} return {"written": True, "subscriber": subscriber, "column": column, "value": value} TOOLS = { "read_meter": {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False}, "consumption": {"function": t_consumption, "fields": ("previous", "current"), "side_effect": False}, "tariff": {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False}, "search_docs": {"function": t_document, "fields": ("query",), "side_effect": False}, "write_record": {"function": t_record, "fields": ("subscriber", "column", "value"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"error": "no such tool"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) # ---- FICTIONAL tasks: each one is solved by a known tool sequence TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("read_meter", "consumption", "tariff")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("read_meter", "consumption")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("search_docs",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("read_meter", "consumption", "tariff", "write_record")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["type"]) # ---- FICTIONAL agent: at each step it picks a tool or finishes def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["approvals"] = state.get("approvals", 0) + 1 if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["result"] = s["m3"] if choice == "tariff": state["result"] = s["amount"] if choice == "search_docs": state["result"] = s["code"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} print("shared environment:", len(SUBSCRIBERS), "subscribers ,", len(DOCUMENTS), "documents ,", len(TOOLS), "tools ,", len(TASKS), "tasks") ``` ``` shared environment: 60 subscribers , 5 documents , 5 tools , 24 tasks ``` #### Three Permission Sets and the Gate The surface opens with three tools: `read_file`, `write_file`, `run_command`; `write_record` in the shared set also declares a write capability. The gate compares this declaration with the permission set and **only stops**; nowhere in the code is there a path that exceeds the limit, and every line that runs is on the side of control. The twenty-four fictional surface tasks are spread over six kinds, and three carry a side effect. ```python # ---- FICTIONAL file tree and terminal surface (specific to this lesson) TREE = {"data/index.csv": 60, "data/result.csv": 0, "docs/tariff.txt": 5, "config/threshold.cfg": 2, "logs/requests.log": 41} COMMANDS = {"index_dump": 60, "log_tail": 12, "archive_cleanup": 7, "meter_reset": 0} PATH_HINT = {"endek": "data/index.csv", "sonuc": "data/result.csv", "tarif": "docs/tariff.txt", "esik": "config/threshold.cfg", "gunlu": "logs/requests.log"} COMMAND_HINT = {"dokum": "index_dump", "kuyru": "log_tail", "temiz": "archive_cleanup", "sifir": "meter_reset"} def resolve_path(a, mode): """Read takes the first match , write takes the last.""" e = [PATH_HINT[t] for t in tokenize(a) if t in PATH_HINT] return (e[0] if mode == "read" else e[-1]) if e else "logs/requests.log" def resolve_command(a): return next((COMMAND_HINT[t] for t in tokenize(a) if t in COMMAND_HINT), None) def t_read_file(query="", subscriber=None, **_): path = resolve_path(query, "read") if path == "data/index.csv": a = SUBSCRIBERS.get(subscriber) return {"error": "missing field"} if a is None else { "previous": a["previous"], "current": a["current"], "type": a["type"]} if path == "docs/tariff.txt": return {"result": "TR01"} return {"result": ABOVE if path == "config/threshold.cfg" else TREE[path]} def t_write_file(query="", result=None, **_): return {"error": "missing field"} if result is None else {"written": resolve_path(query, "write")} def t_command(query="", **_): k = resolve_command(query) return {"error": "no such command"} if k is None else {"result": COMMANDS[k]} TOOLS.update({"read_file": {"function": t_read_file, "fields": ("query", "subscriber"), "side_effect": False}, "write_file": {"function": t_write_file, "fields": ("query", "result"), "side_effect": True}, "run_command": {"function": t_command, "fields": ("query",), "side_effect": True}}) TOOL_NAMES = list(TOOLS) CAPABILITY = {"read_file": "read", "write_file": "write", "write_record": "write", "run_command": "execute"} PERMISSION = {"full": {"read": ("",), "write": ("",), "command": None}, "read-only": {"read": ("",), "write": (), "command": ()}, "allowlisted": {"read": ("data/", "docs/", "config/"), "write": ("data/result",), "command": ("index_dump", "log_tail")}} def allow(name, state, p): """Compares the surface the tool declares with the permission set given.""" g = CAPABILITY.get(name) if g is None: return True if g == "execute": return p["command"] is None or resolve_command(state.get("query", "")) in p["command"] if name == "write_record": return bool(p[g]) return any(resolve_path(state.get("query", ""), g).startswith(o) for o in p[g]) ACTIVE, STOPPED = [PERMISSION["full"]], {"count": 0} _execute = call_tool def call_tool(name, state): """The gate stands in front of the executor; a call that does not pass never reaches the tool.""" if allow(name, state, ACTIVE[0]): return _execute(name, state) STOPPED["count"] += 1 return {"error": "no permission"} # ---- FICTIONAL surface tasks: 24 tasks , six kinds KINDS = [("index", "{a} abonesinin endeks satirini oku ve tuketimi hesapla", ("read_file", "consumption")), ("document", "tarife basamaklarini belgeden oku", ("read_file",)), ("threshold", "ustu birim ucret esik degerini ayardan oku", ("read_file",)), ("write", "{a} abonesinin endeks farkini hesapla ve sonuc dosyasina yaz", ("read_file", "consumption", "write_file")), ("dump", "endeks dokumunu calistir", ("run_command",)), ("cleanup", "arsiv temizligini calistir", ("run_command",))] SURFACE = [] for i in range(24): r = rng(SEED + 6421 * i) sub = f"A{1 + int(r() * 60):03d}" kind, text, seq = KINDS[pick(r(), [0.20, 0.12, 0.14, 0.18, 0.18, 0.18])] SURFACE.append({"name": f"Y{i + 1:02d}", "kind": kind, "subscriber": sub, "request": text.format(a=sub), "sequence": seq}) def surface_correct(g): a = SUBSCRIBERS[g["subscriber"]] return {"index": a["current"] - a["previous"], "write": a["current"] - a["previous"], "document": "TR01", "threshold": ABOVE, "dump": COMMANDS["index_dump"], "cleanup": COMMANDS["archive_cleanup"]}[g["kind"]] def run(permission, seed=SEED): """For a task to be completed the answer must be correct and the step with a side effect must have run.""" ACTIVE[0] = PERMISSION[permission] CALLS[0] = STOPPED["count"] = 0 c = {"correct": [], "completed": [], "steps": 0, "share": {}} for g in SURFACE: before = STOPPED["count"] s = agent(g, seed=seed) d = s["answer"] == surface_correct(g) c["correct"].append(d) c["completed"].append(d and all(any(x == t and not h for x, h in s["trace"]) for t in g["sequence"] if TOOLS[t]["side_effect"])) c["steps"] += s["steps"] c["share"][g["kind"]] = c["share"].get(g["kind"], 0) + STOPPED["count"] - before c.update(calls=CALLS[0], stopped=STOPPED["count"]) return c print("surface task kinds:", {t[0]: sum(g["kind"] == t[0] for g in SURFACE) for t in KINDS}) print("permission set correct A correct B completed executed stopped steps swapped") table = {} for p in PERMISSION: a, b = run(p), run(p, SEED + 1) table[p] = a print(f"{p:<14} {sum(a['correct']) / 24:>9.4f} {sum(b['correct']) / 24:>9.4f}" f" {sum(a['completed']) / 24:>9.4f} {a['calls']:>8} {a['stopped']:>7}" f" {a['steps']:>5} {sum(x != z for x, z in zip(a['completed'], b['completed'])):>7}") print("\nstopped calls by task kind") for p in ("read-only", "allowlisted"): print(f" {p:<13}", {k: v for k, v in table[p]["share"].items() if v}) ``` ``` surface task kinds: {'index': 6, 'document': 3, 'threshold': 3, 'write': 5, 'dump': 4, 'cleanup': 3} permission set correct A correct B completed executed stopped steps swapped full 0.9583 0.9583 0.9583 54 0 78 2 read-only 0.6250 0.6250 0.5000 46 71 129 0 allowlisted 0.8333 0.8333 0.8333 51 20 92 2 stopped calls by task kind read-only {'write': 22, 'cleanup': 19, 'dump': 30} allowlisted {'cleanup': 20} ``` #### Who Is Billed for Stopping The baseline is the first row: the gate stops no call, and **0.9583** of the tasks finish. Lowering the permission to read raises the stopped calls to **71** but drops the completed tasks to **0.5000**; the allowlist stops **20** calls and stays at **0.8333**. The real information is in the last two lines. The 71 stops of read-only are spread over three kinds: `write` 22, `dump` 30, `cleanup` 19. All of the 20 stops of the allowlist, by contrast, are in the `cleanup` kind. This is the measured form of the principle of least privilege: cutting permission by capability drops three task kinds at once, while the allowlist loads the whole cost on the kind it **wants to shut down**. Its difference from full permission is only **0.1250**, that is, three `cleanup` tasks. Whether this difference is measured is told by the last column. The totals of the two runs are the same in all three rows, but a constant total does not mean run-to-run variability is zero: the **swapped** column says that under full permission and under the allowlist **two** tasks each changed their result, and that the total did not move because they canceled each other. The measurement band is $2/24 = 0.0833$; 0.1250 is above the band and is **measured**, and the 0.4583 drop of read-only is far above it. Two readings remain. **The correct answer and the completed task separate under read-only**: 0.6250 correct, 0.5000 completed. In the three tasks in between, the agent produced the right number but **the write call was stopped at the gate**; a measure that looks only at the answer would have counted all three as finished. The steps column says stopping is not free: an error returned from the gate looks to the loop like a tool error, and the agent retries the step. Read-only spends 129 steps instead of 78, that is **65% more**, and finishes less work. #### Summary - The permission surface is the capability a tool declares together with the path or command it touches; the gate compares this declaration with the permission set and only stops. - Under full permission 0.9583 of the 24 tasks finish and no call stops; read-only stops 71 calls and lowers completed tasks to 0.5000; the allowlist stops 20 and stays at 0.8333. - All 20 stops of the allowlist are in a single task kind, while those of read-only are spread over three kinds; this is the measurable difference of the principle of least privilege. - A correct answer does not mean a completed task: under read-only the answer was correct in three tasks but the write was stopped, so the measure reads 0.5000 instead of 0.6250. - Stopping is not free: read-only spends 129 steps, 65% more than the 78 steps of full permission; and because two tasks swapped places in the two runs, the measurement band is 0.0833. #### Next Step In this lesson every surface the agent touched was **ours**: the file tree, the command set, and the gate were all inside the measurement network. Being able to stop a call was possible because we knew who was permitted to do what. When the agent brings in information from outside, this ground shifts: the retrieved text is not ours, it cannot be checked by a permission set, and once it enters the state it is carried to all the steps. The next lesson treats outside information as a **source classification** and counts two things: how the class mix changes the share of correct answers, and how many steps a wrong source is carried. ### Web Search and Browsing Source: https://academia.sh/en/courses/ai-agents/web-search-and-browsing Counting access to outside information as a source classification: eighteen fictional sources are classified as primary, secondary, and unverified, and twenty-four outside-information tasks are run in five setups. The loop without outside information gives 0.2083 correct answers; the primary source alone raises it to 0.7500; adding the secondary class to the mix drops it to 0.3750 and pollutes ten tasks; the three classes together raise the polluted tasks to fifteen. On the same pool, ranking the choice by class brings the correct answer back to 0.7500, lowers the polluted tasks to five, and leaves the call count unchanged. After a wrong code enters the state it is carried for an average of 3.40 to 4.00 steps, and in all of the polluted tasks it reaches the record-writing step. The measurement band is 0.0833, so both the 0.3750 drop and the 0.3750 recovery are measured. The previous lesson tied the surface the agent touches to a permission gate: which path would be read and which command would run was a set that we declared. Outside information removes that ground. When the agent finds a record with a search tool and reads its content with a browsing tool, the text that arrives is **not ours**; it fits neither an allowlist nor a schema. Only one question can be asked of information that comes from outside: **who kept this record.** The lesson turns that question into a **source classification** and counts two things: how the mix moves the share of correct answers, and how many steps a wrong record is carried after it enters the state. - **TM37.** The code is a **simulator**; no real search or browsing tool is called. The sources, the claims, and the questions are **fictional**, seed **20260218**. - **TM38. Sources are named only by their class.** No site, domain name, institution, or country name is written; a source is nothing more than a fictional record number and a class. - **TM39. There are three classes:** **primary** (the operator's own record), **secondary** (a derived compilation), **unverified** (open contribution). The declared source reliability shares are 0.95, 0.70, and 0.30 and are fictional; what is measured is not the value of the share but the result of the mix. - **TM40.** Each of the eighteen fictional sources covers a single topic and on that topic either carries the correct record or does not; a source that does not carries **the code of another topic**. - **TM41.** The twenty-four tasks are spread over six topics; two topics exist in the internal document store, and four are **only outside**. - **TM42. The task sequence is three steps:** search, fetch, write to the record. The wrong code does not stay in the answer, it can **also reach the step with a side effect**. - **TM43. The baseline is the loop without outside information** — the agent looks only at the internal document store. The cost is written as **tool calls, steps, polluted tasks**, and the **steps over which** the wrong code is carried. #### The Shared Loop The environment, the five basic tools, and the agent loop are the same throughout the course. ```python SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL environment: records of the measurement network REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "previous": previous, "current": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1) def t_read_meter(subscriber=None, **_): a = SUBSCRIBERS.get(subscriber) if a is None: return {"error": "no such subscriber"} return {"previous": a["previous"], "current": a["current"], "type": a["type"], "region": a["region"]} def t_consumption(previous=None, current=None, **_): if previous is None or current is None: return {"error": "missing field"} f = current - previous return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, type="residential", **_): if m3 is None: return {"error": "missing field"} return {"amount": _charge(m3, type)} def t_document(query=None, **_): if not query: return {"error": "missing field"} q = set(tokenize(query)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"} def t_record(subscriber=None, column=None, value=None, **_): if not (subscriber and column and value is not None): return {"error": "missing field"} return {"written": True, "subscriber": subscriber, "column": column, "value": value} TOOLS = { "read_meter": {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False}, "consumption": {"function": t_consumption, "fields": ("previous", "current"), "side_effect": False}, "tariff": {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False}, "search_docs": {"function": t_document, "fields": ("query",), "side_effect": False}, "write_record": {"function": t_record, "fields": ("subscriber", "column", "value"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"error": "no such tool"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) # ---- FICTIONAL tasks: each one is solved by a known tool sequence TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("read_meter", "consumption", "tariff")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("read_meter", "consumption")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("search_docs",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("read_meter", "consumption", "tariff", "write_record")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["type"]) # ---- FICTIONAL agent: at each step it picks a tool or finishes def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["approvals"] = state.get("approvals", 0) + 1 if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["result"] = s["m3"] if choice == "tariff": state["result"] = s["amount"] if choice == "search_docs": state["result"] = s["code"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} print("shared environment:", len(SUBSCRIBERS), "subscribers ,", len(DOCUMENTS), "documents ,", len(TOOLS), "tools ,", len(TASKS), "tasks") ``` ``` shared environment: 60 subscribers , 5 documents , 5 tools , 24 tasks ``` #### Source Class, Mix, and Choice Two tools are added to the surface: `web_search` returns the records that match the topic, and `fetch_page` reads the code the chosen record carries. The only design decision the search tool makes is this: **which classes are in the mix and which one is chosen from the candidate list.** The choice does not verify a record, it looks only at its class. ```python # ---- FICTIONAL outside information surface: sources are named by class, there is no place name CODES = ("SY01", "TR01", "BR01", "BR02", "BR03", "BR04") QUESTION = {"SY01": "sayac donusunde endeks farki nasil hesaplanir", "TR01": "tarife basamaklari hangi metrekup degerlerinde degisir", "BR01": "bolge dagitim katsayisi kac", "BR02": "sayac degisim araligi kac yil", "BR03": "itiraz suresi kac gun", "BR04": "gecikme orani nasil hesaplanir"} TOPIC_HINT = {"endek": "SY01", "basam": "TR01", "katsa": "BR01", "degis": "BR02", "itira": "BR03", "gecik": "BR04"} CLASSES = ("primary", "secondary", "unverified") RELIABILITY = {"primary": 0.95, "secondary": 0.70, "unverified": 0.30} SOURCES, FETCHED = {}, [] for i in range(18): r = rng(SEED + 2593 * i) s = CLASSES[pick(r(), [0.28, 0.34, 0.38])] SOURCES[f"K{i + 1:02d}"] = {"class": s, "coverage": CODES[int(r() * 6)], "correct": r() < RELIABILITY[s]} MIX, RANKED = [set(CLASSES)], [False] def resolve_topic(a): return next((TOPIC_HINT[t] for t in tokenize(a) if t in TOPIC_HINT), None) def t_web_search(query="", **_): code = resolve_topic(query) candidates = [k for k, v in SOURCES.items() if v["coverage"] == code and v["class"] in MIX[0]] if not candidates: return {"error": "no source"} if RANKED[0]: # class-ranked choice: primary first candidates.sort(key=lambda k: (CLASSES.index(SOURCES[k]["class"]), k)) return {"source": candidates[0], "found": len(candidates)} def t_fetch_page(source=None, **_): v = SOURCES.get(source) FETCHED.append(None if v is None else v["correct"]) # stays aligned with the trace if v is None: return {"error": "no source"} return {"result": v["coverage"] if v["correct"] else CODES[(CODES.index(v["coverage"]) + 3) % 6], "class": v["class"]} TOOLS.update({"web_search": {"function": t_web_search, "fields": ("query",), "side_effect": False}, "fetch_page": {"function": t_fetch_page, "fields": ("source",), "side_effect": False}}) TOOL_NAMES = list(TOOLS) EXTERNAL = [] for i in range(24): r = rng(SEED + 4643 * i) code = CODES[pick(r(), [0.16, 0.16, 0.18, 0.16, 0.16, 0.18])] EXTERNAL.append({"name": f"W{i + 1:02d}", "code": code, "subscriber": f"A{1 + int(r() * 60):03d}", "request": QUESTION[code], "sequence": ("web_search", "fetch_page", "write_record")}) def run(classes, ranked=False, internal=False, seed=SEED): """internal=True: no outside information, the agent looks only at the internal document store.""" MIX[0], RANKED[0] = set(classes), ranked CALLS[0] = 0 correct = steps = polluted = carried = record = 0 for g in EXTERNAL: FETCHED.clear() s = agent(dict(g, sequence=("search_docs", "write_record")) if internal else g, seed=seed) d = s["answer"] == g["code"] correct, steps = correct + d, steps + s["steps"] place = [i for i, (a, _) in enumerate(s["trace"]) if a == "fetch_page"] wrong = [place[m] for m, ok in enumerate(FETCHED) if ok is False] if wrong and not d: polluted, carried = polluted + 1, carried + s["steps"] - wrong[-1] record += any(a == "write_record" and not h for a, h in s["trace"]) return correct, polluted, carried / polluted if polluted else 0.0, record, CALLS[0], steps print("source class distribution:", {s: sum(v["class"] == s for v in SOURCES.values()) for s in CLASSES}) print("sources of each class carrying correct information:", {s: sum(v["class"] == s and v["correct"] for v in SOURCES.values()) for s in CLASSES}) print("task topic:", {k: sum(g["code"] == k for g in EXTERNAL) for k in CODES}) print("\nsetup correct A correct B polluted tasks carried steps" " polluted records tool calls steps") for name, cl, rk, inn in (("no outside information", (), False, True), ("primary only", ("primary",), False, False), ("primary + secondary", CLASSES[:2], False, False), ("three classes", CLASSES, False, False), ("three classes , ranked", CLASSES, True, False)): a = run(cl, rk, inn) b = run(cl, rk, inn, SEED + 1) print(f"{name:<22} {a[0] / 24:>9.4f} {b[0] / 24:>9.4f} {a[1]:>14}" f" {a[2]:>13.2f} {a[3]:>16} {a[4]:>10} {a[5]:>5}") ``` ``` source class distribution: {'primary': 6, 'secondary': 5, 'unverified': 7} sources of each class carrying correct information: {'primary': 6, 'secondary': 3, 'unverified': 1} task topic: {'SY01': 3, 'TR01': 2, 'BR01': 5, 'BR02': 5, 'BR03': 4, 'BR04': 5} setup correct A correct B polluted tasks carried steps polluted records tool calls steps no outside information 0.2083 0.2083 0 0.00 0 91 110 primary only 0.7500 0.8333 0 0.00 0 121 139 primary + secondary 0.3750 0.4167 10 3.40 10 121 139 three classes 0.3750 0.3750 15 3.60 15 103 126 three classes , ranked 0.7500 0.7917 5 4.00 5 103 126 ``` #### The Number the Mix Changes The baseline row is the loop without outside information: because the internal document store covers two of the six topics, the correct answer stays at **0.2083**. Opening the primary sources raises it to **0.7500** with zero polluted tasks: all six primary sources carry the correct record. The cost is a rise from 91 to **121** calls and from 110 to **139** steps; because the primary class covers four of the six topics, on the remaining two the search returns `no source` and the agent retries the step. The third row is the example of the course's second claim. Adding the secondary class to the mix **drops the correct answer from 0.7500 to 0.3750**. The mechanism is in the second output line: only three of the five secondary sources carry the correct record, and because the choice does not look at the class, a secondary record can come ahead of a primary one. Adding sources here **added noise, not information**. Adding the unverified class too leaves the correct answer at 0.3750 but raises the polluted tasks **from 10 to 15**; in return the calls fall from 121 to **103**, because once coverage is complete there is no `no source` error left. Cheaper and more polluted. The last row shows where the fix is. The pool is the same, the mix is the same, and the calls stay at **103** and the steps at **126**; the only thing that changed is that the candidate list is **ranked by class**. The correct answer returns to **0.7500**, and the polluted tasks fall from 15 to **5**. We had reached the same hit rate with the primary source alone, but there it took 121 calls and 139 steps. **Ranking the classes, not discarding the unreliable class,** is both more correct and cheaper. The five polluted tasks that remain also write down the limit of ranking: the two topics with no primary source at all correspond, in the task distribution, to exactly **five** tasks. The two run columns tell which difference is real: the largest run-to-run difference is between 0.8333 and 0.7500 on the primary-only row, and the band is **0.0833**. Both the 0.3750 drop and the 0.3750 recovery are far above the band and are **measured**. The life of the pollution is in the polluted task and carried step columns: after the wrong code enters the state it is carried for an average of **3.40**, **3.60**, and **4.00** steps, and **all** of the polluted tasks also appear in the polluted record column. The wrong source does not stop at the answer, it becomes permanent through `write_record`. #### Summary - In outside information, what can be checked is not the correctness of the content but the class of the record; sources are counted as primary, secondary, and unverified, and none of them is named. - The loop without outside information gives 0.2083 correct answers; the primary source alone raises it to 0.7500, at the cost of calls rising from 91 to 121. - Adding the secondary class to the mix drops the correct answer to 0.3750; adding sources can add noise instead of information, and this is an example of the second claim of this course. - The three classes together raise the polluted tasks to 15 but lower the calls to 103; on the same pool, ranking the choice by class brings the correct answer to 0.7500, lowers the polluted tasks to 5, and does not raise the calls. - A wrong code is carried for an average of 3.40 to 4.00 steps and reaches the record-writing step in all of the polluted tasks; because the measurement band is 0.0833, both 0.3750 swings are measured. #### Next Step In this lesson the number of steps over which a wrong code is **carried** was counted, but the thing that carries it was not measured. The state of the agent grows at every step: observations accumulate, the trace lengthens, and it is sent again with every call that follows. The context window was defined in the Introduction to AI Engineering course, and resending the conversation state with every request was measured in the Building Applications with Model APIs course; neither is repeated. The next lesson asks what is specific to the loop: how many tokens are carried per step, how many steps fit in the window, and **which field** drops when the window fills. ### Short-Term Memory Source: https://academia.sh/en/courses/ai-agents/short-term-memory Counting step by step the conversation state that the agent loop carries: in a single task the tokens carried are 79 at the first step and rise to 255 at the seventh, and most of the growth comes not from the state but from the trace (at the seventh step the state is 63 tokens and the trace load is 192). Twenty-four tasks are run at five window values. With an unlimited window the correct answer is 0.9583, the total is 9316 tokens, and tool errors are 8. At a window of 200 the correct answer falls to 0.8750, at 150 to 0.8333, at 120 to 0.7083; the dropped fields rise from 0 to 85 and tool errors from 8 to 50. The fields dropped most are the identity fields of the task: abone 16, anahtar 16, sutun 15. Narrowing the window does not lower the tokens carried, it raises them: because steps rise from 97 to 112, the total climbs from 9316 to 12026. The measurement band is 0.0833, so the 0.0833 drop at the window of 200 does not count as a finding. The previous lesson counted that a wrong record, after it enters the state, is carried for an average of three to four steps. What carries it is called **short-term memory**: the fields the agent holds during that task and the trace of the steps that produced those fields. As the loop lengthens this load grows, and at some point it hits a limit. The limit was named in the Introduction to AI Engineering course; that the conversation state is resent with every request, and what this holds per turn, was measured in the Building Applications with Model APIs course. The same limit was counted in the retrieval pipeline too: in the Embeddings and Retrieval-Augmented Generation course an average of 11.3 chunks fit in the window. None of the three routines is repeated. The only thing this lesson asks is specific to the loop: **how many tokens are carried per step, how many steps fit in the window, and which field drops from the state when the window fills.** - **TM44.** The code is a **simulator**; the environment, the tasks, and the tokenizer are **fictional**, seed **20260218**. The window values are fictional too; what is measured is not the value of the threshold but the effect of narrowing. - **TM45. The load carried has two parts:** the **state load**, the tokens of the fields the agent holds; and the **step trace load**, the room the steps so far take up in the loop. The second is the shared loop's own count and grows by 22 tokens per step; in the text it is called briefly the **trace load**. - **TM46. The window is placed on the sum of the state and the trace.** If the total exceeds the threshold, the **field that entered the state first** drops, and the dropping continues; the trace load cannot be dropped. - **TM47. A dropped field is silent.** The agent does not see that a field is gone; the loss shows up only when the tool that asks for that field returns `missing field`. - **TM48. Fitting steps** is the number of calls made until the first drop happens; in a task with no drop, all calls are counted. - **TM49. The baseline is the unlimited window** — the loop in which the limit never binds. The cost is written as **total tokens, steps, tool errors, and dropped fields**. - **TM50.** The task set is the course's shared twenty-four tasks; the measure is again the answer in the final state. #### The Shared Loop The environment, the five basic tools, and the agent loop are the same throughout the course. The tool and field names in the code are the `read_meter`, `consumption`, `tariff`, `search_docs`, and `write_record` tools of the previous lessons; they are counted in tokens here, so they stay Turkish. ```python SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL environment: records of the measurement network # wire words stay Turkish: they are counted in tokens (same rule as the MCP course) # onceki/simdiki/tip/bolge = previous/current/type/region; mesken/isyeri = residential/commercial REGION = ["dogu", "bati", "kuzey", "guney", "merkez"] TYPE = ["mesken", "isyeri"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"bolge": REGION[int(r() * 5)], "tip": TYPE[int(r() * 2)], "onceki": previous, "simdiki": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "isyeri" else 1) def t_read_meter(abone=None, **_): a = SUBSCRIBERS.get(abone) if a is None: return {"hata": "abone yok"} return {"onceki": a["onceki"], "simdiki": a["simdiki"], "tip": a["tip"], "bolge": a["bolge"]} def t_consumption(onceki=None, simdiki=None, **_): if onceki is None or simdiki is None: return {"hata": "eksik alan"} f = simdiki - onceki return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, tip="mesken", **_): if m3 is None: return {"hata": "eksik alan"} return {"tutar": _charge(m3, tip)} def t_document(anahtar=None, **_): if not anahtar: return {"hata": "eksik alan"} q = set(tokenize(anahtar)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"kod": code, "metin": DOCUMENTS[code]} if best > 0 else {"hata": "bulunamadi"} def t_record(abone=None, sutun=None, deger=None, **_): if not (abone and sutun and deger is not None): return {"hata": "eksik alan"} return {"yazildi": True, "abone": abone, "sutun": sutun, "deger": deger} TOOLS = { "sayac_oku": {"function": t_read_meter, "fields": ("abone",), "side_effect": False}, "tuketim": {"function": t_consumption, "fields": ("onceki", "simdiki"), "side_effect": False}, "tarife": {"function": t_tariff, "fields": ("m3", "tip"), "side_effect": False}, "belge_ara": {"function": t_document, "fields": ("anahtar",), "side_effect": False}, "kayit_yaz": {"function": t_record, "fields": ("abone", "sutun", "deger"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"hata": "arac yok"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) # ---- FICTIONAL tasks: each one is solved by a known tool sequence TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("sayac_oku", "tuketim", "tarife")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("sayac_oku", "tuketim")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("belge_ara",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("sayac_oku", "tuketim", "tarife", "kayit_yaz")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["simdiki"] - a["onceki"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["tip"]) # ---- FICTIONAL agent: at each step it picks a tool or finishes def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"abone": g["subscriber"], "anahtar": g["request"], "sutun": "bedel"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("sonuc"), "approvals": state.get("onay", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["onay"] = state.get("onay", 0) + 1 if choice == "kayit_yaz": state["deger"] = state.get("sonuc") s = call_tool(choice, state) trace.append((choice, "hata" in s)) tok += 30 if "hata" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "hata"}) if choice == "tuketim": state["sonuc"] = s["m3"] if choice == "tarife": state["sonuc"] = s["tutar"] if choice == "belge_ara": state["sonuc"] = s["kod"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("sonuc"), "approvals": state.get("onay", 0)} print("shared environment:", len(SUBSCRIBERS), "subscribers ,", len(DOCUMENTS), "documents ,", len(TOOLS), "tools ,", len(TASKS), "tasks") ``` ``` shared environment: 60 subscribers , 5 documents , 5 tools , 24 tasks ``` #### Tokens Carried per Step The measure of short-term memory is not a single number but a **curve**: the load carried at each step is larger than at the previous one. The first block prints this step by step in a single task; the second spreads the same growth over twenty-four tasks at five window values. ```python # ---- Short-term memory: state + trace, which must fit into a window MEMORY = {"window": 10 ** 9, "calls": 0, "tok": 0, "dropped": {}, "fits": -1, "profile": None} _execute = call_tool def state_tokens(d): return sum(len(tokenize(f"{k} : {v}")) for k, v in d.items()) def call_tool(name, state): """If the window is exceeded, the field that entered the state first drops; the trace load cannot be dropped.""" b = MEMORY b["calls"] += 1 trace_load = 60 + 22 * (b["calls"] - 1) while state_tokens(state) + trace_load > b["window"] and len(state) > 1: k = next(iter(state)) b["dropped"][k] = b["dropped"].get(k, 0) + 1 if b["fits"] < 0: b["fits"] = b["calls"] - 1 del state[k] d = state_tokens(state) b["tok"] += d + trace_load if b["profile"] is not None: b["profile"].append((b["calls"], name, len(state), d, trace_load)) return _execute(name, state) def run(window, seed=SEED): MEMORY.update(window=window, tok=0, dropped={}) correct = steps = errors = calls = fits = 0 for g in TASKS: MEMORY.update(calls=0, fits=-1) s = agent(g, seed=seed) correct += s["answer"] == correct_answer(g) steps, errors = steps + s["steps"], errors + s["errors"] calls += MEMORY["calls"] fits += MEMORY["calls"] if MEMORY["fits"] < 0 else MEMORY["fits"] return (correct, errors, steps, MEMORY["tok"], MEMORY["tok"] / calls, fits / 24, sum(MEMORY["dropped"].values()), MEMORY["dropped"]) sample = next(g for g in TASKS if g["kind"] == "record") MEMORY.update(window=10 ** 9, calls=0, tok=0, dropped={}, fits=-1, profile=[]) agent(sample, seed=SEED) print(f"single task ({sample['kind']}) , unlimited window: tokens carried per step") print("step tool fields state trace load total") for c, name, n, d, y in MEMORY["profile"]: print(f"{c:>4} {name:<12} {n:>6} {d:>5} {y:>10} {d + y:>6}") MEMORY["profile"] = None print("\nwindow correct A correct B steps total tokens tokens/call fits errors A" " errors B dropped") for w in (10 ** 9, 260, 200, 150, 120): a, b = run(w), run(w, SEED + 1) print(f"{'unlimited' if w > 10 ** 6 else w:<9} {a[0] / 24:>9.4f} {b[0] / 24:>9.4f}" f" {a[2]:>5} {a[3]:>12} {a[4]:>11.1f} {a[5]:>4.2f} {a[1]:>8} {b[1]:>8}" f" {a[6]:>7}") most = sorted(run(120)[7].items(), key=lambda x: (-x[1], x[0])) print("\nwindow 120 , most dropped fields:", most[:6]) ``` ``` single task (record) , unlimited window: tokens carried per step step tool fields state trace load total 1 sayac_oku 3 19 60 79 2 belge_ara 7 34 82 116 3 belge_ara 10 54 104 158 4 belge_ara 10 54 126 180 5 tuketim 10 54 148 202 6 tarife 11 57 170 227 7 kayit_yaz 13 63 192 255 window correct A correct B steps total tokens tokens/call fits errors A errors B dropped unlimited 0.9583 0.9583 97 9316 125.9 3.08 8 7 0 260 0.9583 0.9583 97 9297 125.6 3.04 9 7 3 200 0.8750 0.9167 98 9681 125.7 2.75 15 10 34 150 0.8333 0.7500 100 9926 124.1 2.42 28 35 40 120 0.7083 0.6250 112 12026 126.6 1.83 50 58 85 window 120 , most dropped fields: [('abone', 16), ('anahtar', 16), ('sutun', 15), ('onceki', 10), ('simdiki', 10), ('bolge', 5)] ``` #### The Field That Drops When the Window Fills The first output says where the growth comes from. The state load is **19** tokens at the first step and **63** at the seventh; the number of fields rises from three to thirteen. The trace load goes from **60** to **192** and at the seventh step is **three times** the state. The total rises from 79 to **255**. Most of the growth comes not from the information the agent has collected, but from **how many steps it has taken**. This distinction shapes the rest of the lesson, because when the window narrows the only part that can be dropped is the state. The second table binds the limit. With an unlimited window **0.9583** of the twenty-four tasks are correct, a total of **9316** tokens is carried, no field drops, and tool errors are **8**. When the window goes down to 260, three fields drop and performance does not move. At 200 the dropped fields rise to **34**, tool errors to **15**, and the correct answer falls to **0.8750**. At 150, dropped 40, errors 28, correct answer 0.8333. At 120, dropped **85**, errors **50**, correct answer **0.7083**. The fits column reads the narrowing directly: from **3.08** to **1.83**, that is, in the narrowest window a field is gone before even two calls are completed on average. Which field goes is in the last line and comes from the layout itself. The ones dropped most are the fields that **entered the state first**: `abone` (subscriber) **16**, `anahtar` (query) **16**, `sutun` (column) **15**; that is, the identity information of the task. Then `onceki` (previous) and `simdiki` (current) drop ten times each. These explain the outcome: when `abone` is gone, `sayac_oku` (read meter) is called without a subscriber, and when `onceki` is gone, `tuketim` (consumption) returns a missing field. The rise of tool errors from 8 to 50 is the count of a single mechanism, and it moves in the same direction in both runs (7, 7, 10, 35, 58). The performance column must be read more carefully. The largest two run-to-run differences are between 0.8333 and 0.7500, and the band is **0.0833**. The drop between the unlimited window and 200 is also **exactly 0.0833**: not below the band but not above it either, so it **does not count as a finding**; the mechanism is visible in the dropped fields and tool errors columns, and its effect on task performance has not been measured. The 0.1250 at 150 and the 0.2500 at 120 are above the band and are **measured**. The steps and total tokens columns are the most counterintuitive result of the lesson. Narrowing the window does **not lower** the tokens carried: the load per call goes from 125.9 to 126.6, that is, it does not change at all. The total, however, rises **from 9316 to 12026**, by 29%, because the dropped fields produce tool errors and the loop takes **112** steps instead of 97 to finish the same work. The constraint itself is not a saving mechanism: the part it shrinks is the state load, and that is already the smaller part of the load; the part that grows, the trace load, is not affected by the constraint at all. #### Summary - The load carried per step has two parts: while the state load rises from 19 to 63 tokens in a single task, the trace load rises from 60 to 192 and at the seventh step is three times the state. - With an unlimited window the correct answer is 0.9583, the total is 9316 tokens, dropped fields are 0, and tool errors are 8. - As the window narrows, the dropped fields rise from 0 to 85, tool errors from 8 to 50, and the fitting calls fall from 3.08 to 1.83. - The dropped fields are the identity fields that entered the state first (abone 16, anahtar 16, sutun 15); the drop is silent and shows up only in the error of the tool that asks for that field. - Narrowing the window does not lower the tokens carried: while the load per call stays constant from 125.9 to 126.6, the total rises from 9316 to 12026 because steps rise from 97 to 112. - The measurement band is 0.0833; the 0.0833 drop at the window of 200 does not count as a finding, and the 0.1250 and 0.2500 at 150 and 120 are measured. #### Next Step In this lesson memory lived only as long as a single task. The window filled, a field dropped, a tool gave an error, and the loop ended; **when it ended, nothing was left behind**. The next task has to learn the same subscriber, the same document code, and the same tariff band from scratch. What short-term memory holds is the working set of the task, and that set is thrown away when the task ends. The next lesson asks about what is lasting: which information **to write** outside the loop, how many records the decision to write accumulates, and how many steps those records save in later tasks. ### Long-Term Memory Source: https://academia.sh/en/courses/ai-agents/long-term-memory Measuring three write policies on the same second set of 48 tasks: making the observation of every step a separate record accumulates 74 records and saves 44 steps, writing a single answer record per task saves 199 steps with 24 records but drops task success from 0.9167 to 0.5833, and the selective policy, which merges the observations in the plan into a single record, saves 118 steps, again with 24 records. What pays off is not the number of records but the unit of the record. Short-term memory is bounded by the window, but the real loss is not at the bound: **the moment the loop ends, that entire context is erased.** The agent reads the meter, computes the index difference, applies the tariff, and gives the answer. When the same subscriber comes back with another question, the meter is read **again**. Long-term memory fills this gap: a set of records that lives independently of the loop and that later tasks can read. The question here is not **what will be brought back**. Chunking, ranking, and retrieval hits were measured in the Embeddings and Retrieval-Augmented Generation course and are not repeated. The question here is **what will be written**: three write policies are run with the same recall mechanism, and for each one the number of stored records is printed next to the steps saved in later tasks. - **TM61.** The code is a **simulator**; no real model endpoint, tool server, or memory service is called. The environment, the tools, and the tasks are **fictional**, seed **20260218**. - **TM62.** Memory is a **list of records**; each record is a searchable text and its fields. Observation and answer records are keyed by the **subscriber code**, document records by the **question text**. - **TM63. Recall is a single read:** the record that shares the most tokens with the request is returned, and on a tie the one written last. - **TM64.** Three write policies: **every step** (the observation of every step is a separate record), **result only** (a single answer record per task), **selective** (the observations that enter the steps of the plan are **merged into a single record**). - **TM65.** Recalled fields are written into the initial state and a step whose output is known is **skipped**; if the plan is fully covered from memory, the loop never starts. - **TM66.** Records are written from the **first set of 24 tasks**; the measurement is made on **48 new questions asked to the same subscribers**. The step budget is 8, and the baseline is the **loop without memory**. #### What Is Left When the Loop Ends The block below sets up the shared definition and asks a single subscriber two separate questions. The tool and field names in the code are the `read_meter`, `consumption`, `tariff`, `search_docs`, and `write_record` tools of the previous lessons; they are counted in tokens here, so they stay Turkish. ```python # FICTIONAL environment and FICTIONAL agent. No real model endpoint, tool server, or memory # service is called. The subscriber records, documents, and tasks are fictional, seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL environment: records of the measurement network # wire words stay Turkish: they are counted in tokens (same rule as the MCP course) # onceki/simdiki/tip/bolge = previous/current/type/region; mesken/isyeri = residential/commercial REGION = ["dogu", "bati", "kuzey", "guney", "merkez"] TYPE = ["mesken", "isyeri"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"bolge": REGION[int(r() * 5)], "tip": TYPE[int(r() * 2)], "onceki": previous, "simdiki": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "isyeri" else 1) def t_read_meter(abone=None, **_): a = SUBSCRIBERS.get(abone) if a is None: return {"hata": "abone yok"} return {"onceki": a["onceki"], "simdiki": a["simdiki"], "tip": a["tip"], "bolge": a["bolge"]} def t_consumption(onceki=None, simdiki=None, **_): if onceki is None or simdiki is None: return {"hata": "eksik alan"} f = simdiki - onceki return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, tip="mesken", **_): if m3 is None: return {"hata": "eksik alan"} return {"tutar": _charge(m3, tip)} def t_document(anahtar=None, **_): if not anahtar: return {"hata": "eksik alan"} q = set(tokenize(anahtar)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"kod": code, "metin": DOCUMENTS[code]} if best > 0 else {"hata": "bulunamadi"} def t_record(abone=None, sutun=None, deger=None, **_): if not (abone and sutun and deger is not None): return {"hata": "eksik alan"} return {"yazildi": True, "abone": abone, "sutun": sutun, "deger": deger} TOOLS = { "sayac_oku": {"function": t_read_meter, "fields": ("abone",), "side_effect": False}, "tuketim": {"function": t_consumption, "fields": ("onceki", "simdiki"), "side_effect": False}, "tarife": {"function": t_tariff, "fields": ("m3", "tip"), "side_effect": False}, "belge_ara": {"function": t_document, "fields": ("anahtar",), "side_effect": False}, "kayit_yaz": {"function": t_record, "fields": ("abone", "sutun", "deger"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"hata": "arac yok"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) # ---- FICTIONAL tasks: each one is solved by a known tool sequence TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("sayac_oku", "tuketim", "tarife")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("sayac_oku", "tuketim")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("belge_ara",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("sayac_oku", "tuketim", "tarife", "kayit_yaz")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["simdiki"] - a["onceki"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["tip"]) # ---- FICTIONAL agent: at each step it picks a tool or finishes def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"abone": g["subscriber"], "anahtar": g["request"], "sutun": "bedel"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("sonuc"), "approvals": state.get("onay", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["onay"] = state.get("onay", 0) + 1 if choice == "kayit_yaz": state["deger"] = state.get("sonuc") s = call_tool(choice, state) trace.append((choice, "hata" in s)) tok += 30 if "hata" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "hata"}) if choice == "tuketim": state["sonuc"] = s["m3"] if choice == "tarife": state["sonuc"] = s["tutar"] if choice == "belge_ara": state["sonuc"] = s["kod"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("sonuc"), "approvals": state.get("onay", 0)} CALLS[0] = 0 g1 = TASKS[4] g2 = dict(g1, name="G05b", kind="consumption", sequence=("sayac_oku", "tuketim"), request=f"{g1['subscriber']} abonesinin bu donem tuketimi kac metrekup") print("task set:", {t: sum(g["kind"] == t for g in TASKS) for t in ("charge", "consumption", "document", "record")}) print(g1["subscriber"], "two separate questions:", [x[0] for x in agent(g1)["trace"]], [x[0] for x in agent(g2)["trace"]], "; tool calls", CALLS[0]) ``` ``` task set: {'charge': 6, 'consumption': 8, 'document': 7, 'record': 3} A026 two separate questions: ['sayac_oku', 'tuketim', 'tarife'] ['sayac_oku', 'tuketim'] ; tool calls 5 ``` The same subscriber's meter was read twice. The first two steps of the second question repeat the first exactly; two of the five tool calls bring no new information. The loss is not a flaw of the loop but **the absence of a lasting place to write**. #### Three Write Policies The only difference of the agent loop is that the initial state comes from memory and a step whose output is known is skipped. ```python PLAN = {"charge": (("sayac_oku", "tuketim", "tarife"), "{sub} abonesinin bu donem bedeli nedir"), "consumption": (("sayac_oku", "tuketim"), "{sub} abonesinin bu donem tuketimi kac metrekup"), "document": (("belge_ara",), "sayac donusunde endeks farki nasil hesaplanir"), "record": (("sayac_oku", "tuketim", "tarife", "kayit_yaz"), "{sub} abonesinin bedelini hesapla ve kayda yaz")} KINDS = ("charge", "consumption", "document", "record") OUTPUTS = {"sayac_oku": ("onceki", "simdiki", "tip", "bolge"), "tuketim": ("m3",), "tarife": ("tutar",), "belge_ara": ("kod",)} RESULT_KEY = {"tuketim": "m3", "tarife": "tutar", "belge_ara": "kod"} def next_set(n=48): """NEW questions asked to the same subscribers; the recipe is identical to TASKS.""" k = [] for i in range(n): r = rng(SEED + 7919 * (i + 101)) sub = TASKS[int(r() * 24)]["subscriber"] t = KINDS[pick(r(), [0.30, 0.22, 0.20, 0.28])] k.append({"name": f"H{i + 1:02d}", "kind": t, "subscriber": sub, "sequence": PLAN[t][0], "request": PLAN[t][1].format(sub=sub)}) return k def recall(memory, request): """Single read: the record sharing the most tokens, on a tie the one written last.""" b, best, record = set(tokenize(request)), 0, None for text, fields in memory: o = len(b & set(tokenize(text))) if o >= best and o > 0: best, record = o, fields return record def agent_b(g, memory, seed=SEED, max_steps=8): """The only difference from the shared loop: the initial state comes from memory and a step whose output is already known is skipped.""" r = rng(seed ^ trace_id(g["name"])) record = recall(memory, g["request"]) c = {"steps": 0, "read": record is not None, "fields": len(record or {}), "observations": [], "on_path": {}} state = {"abone": g["subscriber"], "anahtar": g["request"], "sutun": "bedel"} state.update(record or {}) if "sonuc" in state: return dict(c, answer=state["sonuc"]) step, k, seq = 0, 0, g["sequence"] while k < len(seq) and seq[k] in OUTPUTS and all(x in state for x in OUTPUTS[seq[k]]): state["sonuc"] = state.get(RESULT_KEY.get(seq[k]), state.get("sonuc")) k += 1 if k == len(seq): return dict(c, answer=state.get("sonuc")) while step < max_steps: step += 1 target = seq[k] if k < len(seq) else None if target is None: if r() < 0.88: break choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if choice == "kayit_yaz": state["deger"] = state.get("sonuc") s = call_tool(choice, state) observation = {x: s[x] for x in OUTPUTS.get(choice, ()) if x in s} c["observations"].append(observation) if "hata" in s: continue state.update({kk: vv for kk, vv in s.items() if kk != "hata"}) if choice in RESULT_KEY: state["sonuc"] = s[RESULT_KEY[choice]] if choice == target: c["on_path"].update(observation) k += 1 return dict(c, steps=step, answer=state.get("sonuc")) def run_policy(policy, seed=SEED, n=48): """First the initial set is run and written according to the policy, then the new set.""" memory = [] for g in TASKS: s = agent_b(g, [], seed) key = g["request"] if g["kind"] == "document" else g["subscriber"] write = (s["observations"] if policy == "every step" else [s["on_path"]] if policy == "selective" else [{"sonuc": s["answer"]}] if policy == "result only" else []) memory += [(key + " " + " ".join(f"{x} {v}" for x, v in a.items()), a) for a in write if None not in a.values()] CALLS[0] = 0 steps = correct = reads = hits = fields = 0 for g in next_set(n): s = agent_b(g, memory, seed) d = s["answer"] == correct_answer(g) steps, correct, reads = steps + s["steps"], correct + d, reads + s["read"] fields, hits = fields + s["fields"], hits + (d and s["read"]) return (len(memory), reads, fields, hits / max(reads, 1), steps, CALLS[0], correct / n) POLICIES = ("memoryless", "every step", "result only", "selective") print("policy records reads fields hit rate steps calls correct | 2nd run") for p in POLICIES: a, b = run_policy(p), run_policy(p, SEED + 1) print(f"{p:<14} {a[0]:>7} {a[1]:>5} {a[2]:>6} {a[3]:>8.4f} {a[4]:>5} {a[5]:>5}" f" {a[6]:>8.4f} | {b[4]:>4} steps {b[6]:.4f}") ``` ``` policy records reads fields hit rate steps calls correct | 2nd run memoryless 0 0 0 0.0000 212 166 0.9167 | 222 steps 0.8958 every step 74 47 99 0.8936 168 132 0.8958 | 176 steps 0.8542 result only 24 45 45 0.5556 13 10 0.5833 | 14 steps 0.6250 selective 24 47 166 0.8511 94 74 0.8542 | 96 steps 0.8958 ``` The baseline finishes the 48 tasks with 212 steps and 166 tool calls, and task success is 0.9167. The three policies are built on top of this row. **Result only** is the most striking row. With 24 records it brings 212 steps down to **13**, which saves 199 steps; tool calls fall from 166 to 10. But task success drops to **0.5833**. The reason is this: the record is an **answer** keyed by the subscriber code, whereas an answer belongs to a question. The subscriber's consumption answer is stored, the next task asks for the same subscriber's **charge**; the value is right, the question is different. The **every step** policy accumulates 74 records, three times as many, and saves only 44 steps (from 212 to 168). The reason is visible in the reads column: the 47 reads bring a total of **99** fields, two fields per read. Because the observation of each step is a separate record, a single read brings the output of **a single step**. The **selective** policy saves 118 steps with the same 24 records (from 212 to 94) and brings **166 fields** in 47 reads; because the observations that enter the plan merge into a single record, one read carries all the preconditions. #### Not the Number of Records but the Unit of the Record The two rankings in the table do not coincide. By the number of records the most generous policy is every step, by steps saved it is result only; the selective policy saves **2.68 times** as many steps as every step with a third of its records. What decides is not how many records are written but **which unit** a record covers: one step, one task, or all the fields of one entity. The second run says which difference counts as a finding. The baseline gives 0.9167 and 0.8958, a run-to-run difference of **0.0417**. The loss of result only is 0.3334 and 0.2708 in the two runs, both far above the band and **measured**. The loss of the selective policy is 0.0625 in one run and **0.0000** in the other; it oscillates around the band and **counts as unmeasured**. That the gain of 118 steps has a cost in task success could not be shown in this setup. The rule is plain: writing **an answer that depends on the question** into memory makes that answer wrong when the question changes. What should be written is the **observation**, which is independent of the question: a meter reading, a subscriber type, a document code. Reproducing the answer at every task is cheaper than a wrong answer. #### Summary - When the loop ends, short-term memory is erased; the same subscriber's meter is read again in the next question. - What is measured is not retrieval but **the write decision**: with the same recall mechanism, three policies give three different results. - Result only saves 199 steps with 24 records but drops task success from 0.9167 to 0.5833; the loss is eight times the run-to-run difference and is measured. - Every step saves 44 steps with 74 records, selective saves 118 steps with 24; the fields brought per read are 99 against 166. - What decides is the unit of the record, not the number of records; writing an observation instead of an answer that depends on the question removes the wrong answer. #### Next Step This lesson treated memory **between** tasks. **Within** a task the problem reverses: the context is not erased, it grows, and when the window fills something has to be thrown out. The decision to discard is the mirror of the decision to write. The next lesson measures two ways of compression: summarizing the observations and keeping only the relevant ones. As the compression ratio rises, the tokens carried fall, but every field that drops may be the input of a next step. These two are what will be measured: the tokens gained and **the number of tasks that need the discarded field**. ### Context Compression Source: https://academia.sh/en/courses/ai-agents/context-compression Sweeping two compaction policies over the same 24 tasks: when the carried state is paid for as well, the baseline gives 21913 tokens and 0.9583 task success. When selective recall cuts the kept fields to three, tokens fall to 16569 and task success does not change at all, but the number of tasks that need the dropped field rises from 0 to 8. Summarization falls to 0.7917 on the same budget and at the tightest setting raises tokens to 33566, 1.53 times the baseline. This is the point where compaction makes the job worse. The previous lesson treated memory between tasks: what was written determined the steps saved in later tasks. Within a task the problem reverses. The context is not erased, it **grows**: every observation is added to the state and carried again at every later step. The token count of the shared loop sees only half of this: it counts the step trace, it does not count the carried state itself. This lesson first adds that missing item, then sweeps two policies of **context compaction**: **summarization** drops the oldest fields and puts a single summary entry in their place; **selective recall** keeps the fields that the remaining steps declare. What will be measured is two numbers side by side: as the compaction ratio rises, the carried tokens fall, but **the number of tasks that need the dropped field** grows. This lesson is an instance of the course's second claim; at the tightest setting compaction raises tokens **above** the baseline. - **TM67.** The code is a **simulator**; no real model endpoint or tool server is called. The environment, the tools, and the tasks are **fictional**, seed **20260218**. - **TM68.** The carried state is a token item: each field is paid 12 tokens for its name and format and its own token count for its value, and this cost is paid **again at every step**. - **TM69. The compaction ratio is given by the `keep` setting:** at most `keep` fields are carried in the state and the excess is dropped. Compaction is applied at the end of every step. - **TM70. Summarization** drops the oldest fields and puts a single summary entry in their place; the entry takes up the place of a field but carries no usable value. - **TM71. Selective recall** keeps, on the same `keep` budget, the fields that the remaining steps of the plan **declare**; the newest fields fill the leftover room. - **TM72.** When a tool asks for a field that was dropped earlier, a **missing field** error returns. A task in which this error is seen at least once counts as a **task that needs the dropped field**. - **TM73.** The task set is 24, the step budget is 8, and the baseline is the **loop without compaction**. #### The Cost of the Carried State The block below sets up the shared definition and prints the baseline. The tool and field names in the code are the `read_meter`, `consumption`, `tariff`, `search_docs`, and `write_record` tools of the previous lessons; they are counted in tokens here, so they stay Turkish. ```python # FICTIONAL environment and FICTIONAL agent. No real model endpoint, tool server, or memory # service is called. The subscriber records, documents, and tasks are fictional, seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL environment: records of the measurement network # wire words stay Turkish: they are counted in tokens (same rule as the MCP course) # onceki/simdiki/tip/bolge = previous/current/type/region; mesken/isyeri = residential/commercial REGION = ["dogu", "bati", "kuzey", "guney", "merkez"] TYPE = ["mesken", "isyeri"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"bolge": REGION[int(r() * 5)], "tip": TYPE[int(r() * 2)], "onceki": previous, "simdiki": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "isyeri" else 1) def t_read_meter(abone=None, **_): a = SUBSCRIBERS.get(abone) if a is None: return {"hata": "abone yok"} return {"onceki": a["onceki"], "simdiki": a["simdiki"], "tip": a["tip"], "bolge": a["bolge"]} def t_consumption(onceki=None, simdiki=None, **_): if onceki is None or simdiki is None: return {"hata": "eksik alan"} f = simdiki - onceki return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, tip="mesken", **_): if m3 is None: return {"hata": "eksik alan"} return {"tutar": _charge(m3, tip)} def t_document(anahtar=None, **_): if not anahtar: return {"hata": "eksik alan"} q = set(tokenize(anahtar)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"kod": code, "metin": DOCUMENTS[code]} if best > 0 else {"hata": "bulunamadi"} def t_record(abone=None, sutun=None, deger=None, **_): if not (abone and sutun and deger is not None): return {"hata": "eksik alan"} return {"yazildi": True, "abone": abone, "sutun": sutun, "deger": deger} TOOLS = { "sayac_oku": {"function": t_read_meter, "fields": ("abone",), "side_effect": False}, "tuketim": {"function": t_consumption, "fields": ("onceki", "simdiki"), "side_effect": False}, "tarife": {"function": t_tariff, "fields": ("m3", "tip"), "side_effect": False}, "belge_ara": {"function": t_document, "fields": ("anahtar",), "side_effect": False}, "kayit_yaz": {"function": t_record, "fields": ("abone", "sutun", "deger"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"hata": "arac yok"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) # ---- FICTIONAL tasks: each one is solved by a known tool sequence TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("sayac_oku", "tuketim", "tarife")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("sayac_oku", "tuketim")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("belge_ara",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("sayac_oku", "tuketim", "tarife", "kayit_yaz")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["simdiki"] - a["onceki"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["tip"]) # ---- FICTIONAL agent: at each step it picks a tool or finishes def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"abone": g["subscriber"], "anahtar": g["request"], "sutun": "bedel"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("sonuc"), "approvals": state.get("onay", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["onay"] = state.get("onay", 0) + 1 if choice == "kayit_yaz": state["deger"] = state.get("sonuc") s = call_tool(choice, state) trace.append((choice, "hata" in s)) tok += 30 if "hata" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "hata"}) if choice == "tuketim": state["sonuc"] = s["m3"] if choice == "tarife": state["sonuc"] = s["tutar"] if choice == "belge_ara": state["sonuc"] = s["kod"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("sonuc"), "approvals": state.get("onay", 0)} CALLS[0] = 0 t = [agent(g) for g in TASKS] print("baseline: steps", sum(x["steps"] for x in t), ", calls", CALLS[0], ", trace tokens", sum(x["tokens"] for x in t), ", errors", sum(x["errors"] for x in t), ", success", round(sum(x["answer"] == correct_answer(g) for x, g in zip(t, TASKS)) / 24, 4)) ``` ``` baseline: steps 97 , calls 74 , trace tokens 11978 , errors 8 , success 0.9583 ``` The shared loop finishes the 24 tasks in 97 steps and counts 11978 tokens. This number covers **only the trace**: at every step the tool names and observation headings up to that point. Yet the prompt that goes to the model carries the state as well: the index readings that were read, the subscriber type, the document text that was found. Every field is paid again at every step, and as the state grows this item grows faster than the trace. #### Sweeping the Compaction Ratio The block below completes the cost and sweeps the two policies over four settings. ```python NAME_COST = 12 # name and format cost of a carried field RESULT_KEY = {"tuketim": "m3", "tarife": "tutar", "belge_ara": "kod"} def state_tokens(state): return sum(NAME_COST + len(tokenize(v)) for v in state.values()) def compact(state, remaining, keep, policy): """Anything beyond keep fields is dropped. Summary drops the oldest and puts a single summary entry in its place; selective keeps the fields the remaining steps declare.""" if len(state) <= keep: return state, () keys = list(state) if policy == "selective": needed = {a for name in remaining for a in TOOLS[name]["fields"]} | {"sonuc"} keys.sort(key=lambda k: k in needed) dropped = tuple(keys[:len(state) - keep]) new = {k: v for k, v in state.items() if k not in dropped} if policy == "summary": new["ozet"] = f"{len(dropped)} alan dusuruldu" return new, dropped def agent_c(g, keep=99, policy="summary", seed=SEED, max_steps=8): """The shared loop, with context compaction at the end of every step.""" r = rng(seed ^ trace_id(g["name"])) state = {"abone": g["subscriber"], "anahtar": g["request"], "sutun": "bedel"} trace, step, tok, errors, dropped, needed = [], 0, 0, 0, set(), 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) + state_tokens(state) target = seq[k] if k < len(seq) else None if target is None: if r() < 0.88: break choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if choice == "kayit_yaz": state["deger"] = state.get("sonuc") missing = [a for a in TOOLS[choice]["fields"] if a not in state and a in dropped] s = call_tool(choice, state) trace.append(choice) tok += 30 if "hata" in s: errors, needed = errors + 1, needed + bool(missing) continue state.update({kk: vv for kk, vv in s.items() if kk != "hata"}) if choice in RESULT_KEY: state["sonuc"] = s[RESULT_KEY[choice]] if choice == target: k += 1 state, dr = compact(state, seq[k:], keep, policy) dropped |= set(dr) return {"steps": step, "tokens": tok, "errors": errors, "needed": needed, "answer": state.get("sonuc")} def run(keep, policy, seed=SEED): CALLS[0] = 0 st = tok = errors = needed = correct = 0 for g in TASKS: s = agent_c(g, keep, policy, seed) st, tok, errors = st + s["steps"], tok + s["tokens"], errors + s["errors"] needed, correct = needed + (s["needed"] > 0), correct + (s["answer"] == correct_answer(g)) return st, CALLS[0], tok, errors, needed, correct / len(TASKS) SWEEP = [(99, "uncompacted")] + [(t, p) for p in ("summary", "selective") for t in (6, 5, 4, 3)] print("keep policy steps calls tokens errors needed success | 2nd run") for keep, pol in SWEEP: a = run(keep, "summary" if keep == 99 else pol) b = run(keep, "summary" if keep == 99 else pol, SEED + 1) print(f"{keep if keep != 99 else '--':>4} {pol:<14} {a[0]:>5} {a[1]:>5} {a[2]:>8}" f" {a[3]:>6} {a[4]:>6} {a[5]:>8.4f} | {b[2]:>6} {b[5]:.4f}") ``` ``` keep policy steps calls tokens errors needed success | 2nd run -- uncompacted 97 74 21913 8 0 0.9583 | 21998 0.9583 6 summary 102 82 21281 29 5 0.9167 | 23749 0.9167 5 summary 98 77 19027 24 6 0.9583 | 21458 0.9583 4 summary 98 77 18086 24 6 0.7917 | 20420 0.7500 3 summary 156 149 33566 125 18 0.2917 | 34523 0.2500 6 selective 97 74 19023 15 3 0.9583 | 19233 1.0000 5 selective 97 74 18004 18 5 0.9583 | 18184 1.0000 4 selective 98 77 17475 24 6 0.9583 | 18846 1.0000 3 selective 98 77 16569 24 8 0.9583 | 17861 1.0000 ``` Once the state item is added, the baseline rises from 11978 to **21913 tokens**: the carried state holds more than the trace itself. This is where compaction belongs. The selective rows advance as expected. As `keep` goes from six to three, tokens fall from 19023 to **16569**, that is, by **0.2439** relative to the baseline. The cost shows in two columns: tool errors climb from 8 in the uncompacted row to 24, and the **number of tasks that need the dropped field** goes from 0 to **8**, that is, climbs to a third of the twenty-four tasks. Task success stays fixed at 0.9583 in the first run and is 1.0000 in the second. The dropped fields break the next step, but the agent gets the field back by calling the same tool again; the cost is paid not in steps but in errors. The summary rows start in the same direction and then break. At `keep` four the tokens, 18086, are a little above the selective ones, but task success is **0.7917**, and 0.7500 in the second run. At `keep` three the table turns around: tokens are **33566**, that is, **1.53 times** the uncompacted baseline; steps climb from 97 to 156, tool calls from 74 to 149, tool errors from 8 to **125**, and task success falls to 0.2917. Here compaction is not a saving but a multiplier: a dropped field leads to an error, the error to a new step, and the new step to a context that is carried again. #### Same Budget, Different Content The most telling place in the table is the `keep` four rows. The two policies carry **the same number of fields**; the token difference is 611, less than four percent. The task success difference, however, is 0.7917 against 0.9583, and in the second run 0.7500 against 1.0000. With the largest difference between runs at 0.0417, this gap is four and six times the band, that is, **measured**. What decides is not the **size** of the context but **which fields remain**. This is the memory-side counterpart of the finding measured in the Building Applications with Model APIs course: there, in an editing task, the expected acceptance was 0.6921 when all files were put into the context and **0.8233** when only the relevant files were selected. The selection there was over files, the selection here is over the agent's own observations; the result points the same way. Irrelevant context is not only expensive, it also takes the place of the relevant. #### Summary - The carried state is paid again at every step: the trace tokens are 11978, and with the state the baseline rises to 21913 tokens. - Selective recall cuts tokens to 16569 at `keep` three, by 0.2439 relative to the baseline, and does not change task success. - In the same sweep the number of tasks that need the dropped field rises from 0 to 8 and tool errors from 8 to 24: the cost of the tokens gained is written here. - On the same `keep` budget summarization gives 0.7917 and selective recall 0.9583; the gap is four and six times the run-to-run difference. - At `keep` three summarization raises tokens to 33566, 1.53 times the baseline; at this point compaction is not a saving but a multiplier. #### Next Step In these two lessons memory was treated from two directions: what to write between tasks, what to carry within a task. In both, the records were assumed to be **correct**. Yet a meter is read again, a tariff changes, a document is renewed; the record in long-term memory stays where it is and **silently turns wrong**. Dropping a field does not solve this, because the problem is not the narrowness of the place but the age of the record. The next lesson sweeps the expiry threshold and puts two numbers side by side: as the threshold tightens, stale records drop, but the share of records that are discarded although they are still valid grows. ### Memory Decay Source: https://academia.sh/en/courses/ai-agents/memory-decay Sweeping the expiry threshold over seven settings: in memory without a threshold 0.4528 of the reads are stale and task success is 0.6389. As the threshold tightens, the stale share falls to 0.0000 and task success rises to 0.9306, but the share of dropped records that are still valid climbs from 0.0000 to 0.9701 and steps rise from 138 to 315. The task success gained by tightening the threshold below eight stays as large as the run-to-run difference itself, that is, it counts as unmeasured. The previous two lessons treated memory from two directions: what to write between tasks, what to carry within a task. In both, the records were assumed to be **correct**. Yet the measurement network moves: a meter is read again in the new period, the index difference turns into another number, the charge changes. The record in memory stays where it is and **turns wrong** without giving any error. The remedy for this is not dropping a field; the problem is not the narrowness of the place but the **age** of the record. **Memory expiry** puts an age limit on a record: a record that exceeds the limit is dropped. This lesson sweeps that limit and prints two numbers side by side. As the threshold tightens, the share of **stale records** falls, but the share of dropped records that are **still valid** grows; the two move in opposite directions with the same setting. - **TM74.** The code is a **simulator**; no real memory service or tool server is called. The environment, the tools, and the tasks are **fictional**, seed **20260218**. - **TM75. The environment moves:** once every six tasks, twelve subscribers get **a new period reading** (the previous index is replaced by the current one). The document vocabulary does not change. - **TM76.** Memory is written with the selective policy: the observations that enter the plan are merged into a single record, the record is keyed by the **subscriber code** (document records by the question text), and it carries **the task order in which it was written**. - **TM77. The expiry threshold** is the greatest age a record may carry; a record whose age is greater than the threshold is dropped at the start of every task. - **TM78. A stale record** is a record whose fields do not coincide with the environment's value at that moment; the **stale share** is counted per read. - **TM79. A lost record** is a record that the threshold dropped but that was **still valid** at the moment it was dropped; the **lost share** is counted per dropped record. - **TM80.** The stream is 72 tasks, the step budget is 8; the baseline is **memory without a threshold**. #### The Record Stays, the Value Changes The block below sets up the shared definition and shows staleness on a single subscriber. ```python # FICTIONAL environment and FICTIONAL agent. No real model endpoint, tool server, or memory # service is called. The subscriber records, documents, and tasks are fictional, seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL environment: records of the measurement network REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "previous": previous, "current": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1) def t_read_meter(subscriber=None, **_): a = SUBSCRIBERS.get(subscriber) if a is None: return {"error": "no such subscriber"} return {"previous": a["previous"], "current": a["current"], "type": a["type"], "region": a["region"]} def t_consumption(previous=None, current=None, **_): if previous is None or current is None: return {"error": "missing field"} f = current - previous return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, type="residential", **_): if m3 is None: return {"error": "missing field"} return {"amount": _charge(m3, type)} def t_document(query=None, **_): if not query: return {"error": "missing field"} q = set(tokenize(query)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"} def t_record(subscriber=None, column=None, value=None, **_): if not (subscriber and column and value is not None): return {"error": "missing field"} return {"written": True, "subscriber": subscriber, "column": column, "value": value} TOOLS = { "read_meter": {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False}, "consumption": {"function": t_consumption, "fields": ("previous", "current"), "side_effect": False}, "tariff": {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False}, "search_docs": {"function": t_document, "fields": ("query",), "side_effect": False}, "write_record": {"function": t_record, "fields": ("subscriber", "column", "value"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"error": "no such tool"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) # ---- FICTIONAL tasks: each one is solved by a known tool sequence TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("read_meter", "consumption", "tariff")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("read_meter", "consumption")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("search_docs",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("read_meter", "consumption", "tariff", "write_record")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["type"]) # ---- FICTIONAL agent: at each step it picks a tool or finishes def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["approvals"] = state.get("approvals", 0) + 1 if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["result"] = s["m3"] if choice == "tariff": state["result"] = s["amount"] if choice == "search_docs": state["result"] = s["code"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} sub = "A026" old = dict(SUBSCRIBERS[sub]) record = t_read_meter(subscriber=sub) SUBSCRIBERS[sub]["previous"] = SUBSCRIBERS[sub]["current"] SUBSCRIBERS[sub]["current"] += 31 # new period reading print("record in memory :", record["previous"], record["current"], "-> consumption", t_consumption(previous=record["previous"], current=record["current"])["m3"]) print("value in environment:", SUBSCRIBERS[sub]["previous"], SUBSCRIBERS[sub]["current"], "-> consumption", t_consumption(previous=SUBSCRIBERS[sub]["previous"], current=SUBSCRIBERS[sub]["current"])["m3"]) SUBSCRIBERS[sub].update(old) ``` ``` record in memory : 88422 88477 -> consumption 55 value in environment: 88477 88508 -> consumption 31 ``` The record did not change, the environment did. The index pair that comes back from memory gives 55 cubic meters, the meter's value at that moment is 31. The agent gets no error, the tool returns no error, the loop gets shorter, and the answer is wrong: a stale record **cannot be told apart** from a correct one. #### Tightening the Threshold The block below sets up the stream, the period renewals, and the threshold sweep. ```python # ---- Expiry: age and validity of a record PLAN = {"charge": (("read_meter", "consumption", "tariff"), "{sub} abonesinin bu donem bedeli nedir"), "consumption": (("read_meter", "consumption"), "{sub} abonesinin bu donem tuketimi kac metrekup"), "document": (("search_docs",), "sayac donusunde endeks farki nasil hesaplanir"), "record": (("read_meter", "consumption", "tariff", "write_record"), "{sub} abonesinin bedelini hesapla ve kayda yaz")} OUTPUTS = {"read_meter": ("previous", "current", "type", "region"), "consumption": ("m3",), "tariff": ("amount",), "search_docs": ("code",)} RESULT_KEY = {"consumption": "m3", "tariff": "amount", "search_docs": "code"} INITIAL = {k: dict(v) for k, v in SUBSCRIBERS.items()} def stream(n=72): """Task stream of n questions asked to the subscribers of the first set.""" k = [] for i in range(n): r = rng(SEED + 7919 * (i + 101)) sub = TASKS[int(r() * 24)]["subscriber"] t = list(PLAN)[pick(r(), [0.30, 0.22, 0.20, 0.28])] k.append({"name": f"H{i + 1:02d}", "kind": t, "subscriber": sub, "sequence": PLAN[t][0], "request": PLAN[t][1].format(sub=sub)}) return k def new_period(i): """Once every six tasks, twelve subscribers get a new period reading.""" r = rng(SEED + 61 * i) for _ in range(12): a = SUBSCRIBERS[f"A{1 + int(r() * 60):03d}"] a["previous"] = a["current"] a["current"] = a["previous"] + int(r() * 70) def is_valid(rec): """Do the record's fields still coincide with the environment's values at this moment.""" if rec["subscriber"] is None: return True a = SUBSCRIBERS[rec["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f now = dict(a, m3=m3, amount=_charge(m3, a["type"])) return all(now.get(x) == v for x, v in rec["fields"].items()) def agent_e(g, rec, seed=SEED, max_steps=8): """The shared loop; the initial state comes from the record in memory.""" r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} state.update(rec["fields"] if rec else {}) step, k, seq, on_path = 0, 0, g["sequence"], {} while k < len(seq) and seq[k] in OUTPUTS and all(x in state for x in OUTPUTS[seq[k]]): state["result"] = state.get(RESULT_KEY.get(seq[k]), state.get("result")) k += 1 if k == len(seq): return {"steps": 0, "answer": state.get("result"), "on_path": {}} while step < max_steps: step += 1 target = seq[k] if k < len(seq) else None if target is None: if r() < 0.88: break choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) if "error" in s: continue state.update({x: v for x, v in s.items() if x != "error"}) if choice in RESULT_KEY: state["result"] = s[RESULT_KEY[choice]] if choice == target: on_path.update({x: s[x] for x in OUTPUTS.get(choice, ()) if x in s}) k += 1 return {"steps": step, "answer": state.get("result"), "on_path": on_path} def run(threshold, seed=SEED, n=72): """Expiry threshold: a record whose age is greater than the threshold is dropped.""" for x, v in INITIAL.items(): SUBSCRIBERS[x].update(v) CALLS[0] = 0 memory = {} s = dict.fromkeys(("reads", "stale", "dropped", "lost", "steps", "correct"), 0) for i, g in enumerate(stream(n)): if i and i % 6 == 0: new_period(i) for key in [x for x, k in memory.items() if i - k["time"] > threshold]: s["dropped"] += 1 s["lost"] += is_valid(memory.pop(key)) key = g["request"] if g["kind"] == "document" else g["subscriber"] rec = memory.get(key) if rec: s["reads"] += 1 s["stale"] += not is_valid(rec) c = agent_e(g, rec, seed) s["steps"] += c["steps"] s["correct"] += c["answer"] == correct_answer(g) if c["on_path"]: memory[key] = {"fields": c["on_path"], "time": i, "subscriber": None if g["kind"] == "document" else g["subscriber"]} return (s["reads"], s["stale"] / max(s["reads"], 1), s["dropped"], s["lost"] / max(s["dropped"], 1), s["steps"], CALLS[0], s["correct"] / n) print("threshold reads stale dropped lost steps calls success | 2nd run") for e in (10 ** 6, 32, 24, 16, 8, 4, 1): a, b = run(e), run(e, SEED + 1) print(f"{e if e < 100 else '--':>9} {a[0]:>6} {a[1]:>7.4f} {a[2]:>9} {a[3]:>7.4f}" f" {a[4]:>7} {a[5]:>7} {a[6]:>9.4f} | {b[6]:.4f}") ``` ``` threshold reads stale dropped lost steps calls success | 2nd run -- 53 0.4528 0 0.0000 138 108 0.6389 | 0.5972 32 42 0.3333 18 0.3333 176 139 0.7778 | 0.7361 24 36 0.2222 23 0.4783 201 158 0.8194 | 0.7778 16 31 0.0968 32 0.6250 220 171 0.8750 | 0.8611 8 23 0.0870 41 0.7805 246 193 0.8889 | 0.8611 4 13 0.0000 55 0.8909 283 221 0.9028 | 0.8750 1 3 0.0000 67 0.9701 315 246 0.9306 | 0.8889 ``` The row without a threshold is the most efficient and the most wrong state of memory: 53 reads, 138 steps, but **0.4528 of the reads are stale** and task success is **0.6389**; in more than a third of the tasks the user is told last period's number. As the threshold tightens, two columns move in opposite directions. The stale share falls from 0.4528 to 0.3333, to 0.2222, to 0.0968, and to **0.0000**, and task success rises from 0.6389 to **0.9306**. In the same rows the lost share climbs from 0.0000 to 0.6250 and to **0.9701**: when the threshold falls to one, 65 of the 67 dropped records were **still valid** at the moment they were dropped. The cost is written in steps as well: 138 steps go to 315, and tool calls from 108 to 246. #### Age Is a Proxy for Validity The lost share column alone explains what expiry is. The threshold does not ask whether a record **is valid**; it asks **how many tasks ago it was written**. Age is a proxy for validity, and in this stream it is a weak one: only twelve subscribers are renewed every six tasks, so staleness is a probability, not a certainty determined by age. The two runs say where to stop. The row without a threshold gives 0.6389 and 0.5972 in the two runs; the band is **0.0417**. The gain of tightening the threshold to eight is 0.2500, six times the band, and is **measured**. The gain of tightening from eight to one, however, lies between 0.8889 and 0.9306, **exactly the band itself**; in the second run the same gain falls to 0.0278. This difference **cannot be declared** a finding: lowering the threshold below eight makes one pay 69 more steps and 53 more tool calls and gives in return a gain that is **unmeasured**. Whoever wants to measure it should enlarge the number of tasks or sweep the renewal frequency as a setting. The stale and lost shares come out exactly the same in the two runs: these two numbers depend not on the tool-selection draw but on the relation between the renewal calendar and the threshold. The expiry threshold cannot be chosen independently of how often the data changes. #### Summary - A record in memory turns wrong without giving an error and cannot be told apart from a correct record. - In memory without a threshold 0.4528 of the reads are stale and task success is 0.6389; when the threshold falls to one, the stale share is 0.0000 and task success is 0.9306. - In the same sweep the share of dropped records that are still valid rises from 0.0000 to 0.9701 and steps climb from 138 to 315. - The threshold asks for the age of a record, not its validity; age is a proxy, and on this renewal calendar the price of zeroing the stale share is throwing away nearly all of the valid records. - The task success gained by tightening the threshold below eight is as large as the run-to-run difference itself and counts as unmeasured; going from eight to one makes one pay 69 more steps. #### Next Step This topic equipped the agent: the tool definition, execution, code execution, the file and terminal surface, outside information, short-term and long-term memory, compaction, and expiry. Together with the previous topic, an agent was built in sixteen lessons, where it would stop was written down, checkpoints were placed, and tools were put in its hands. In every lesson the number and the cost of one mechanism were printed, but each was read with a measure that looked only at the mechanism of that lesson. The question that looks at the whole was never asked: **how do we know this agent works**. Whether a run that gives the correct answer followed the correct **path**, whether a defect of a tool that is not visible when it is tested alone appears inside the flow, and which record can speak about a loop that runs in production were not addressed. The next topic opens with this question. ## Agent Quality Measuring how one knows that a built agent works: the four-state table of the right sequence and the right answer, with a right answer produced by a wrong sequence counted separately; step efficiency read as steps per correct answer; the share of defects placed in five tools that tool-level testing catches, and the class of defects that is never seen; what the end-to-end flow test catches of what the tool level misses, and the residual that neither of the two layers catches; at the result only, tool names, and full observation trace levels, the share of failed tasks whose cause can be found, together with the stored token; the point where cost per step and cost per correct answer split the ranking; and a comparison of the recovery policies of stopping, retrying the same tool, and skipping to another tool by task success, steps, and tool errors. ### Measuring Task Success and Step Efficiency Separately Source: https://academia.sh/en/courses/ai-agents/measuring-task-success-and-step-efficiency-separately Showing on steps that a single success rate stacks four separate states: on a fictional set of twenty-four tasks the toolless single-step baseline comes out at 0.2917, and all of that rate sits in the wrong sequence, right answer box, because all seven document tasks are answered without any tool being called. The loop gives 0.8750 at the narrow budget and 0.9583 at the course budget; at the course budget the only failure is in the right sequence, wrong answer box and comes from a task whose sequence was completed in full being spoiled by an extra call in between. Steps per correct answer come out at 3.4286 for the baseline and 4.2174 at the course budget, so read alone, efficiency rewards the baseline that calls no tool; on the seventeen tasks that need tools the baseline gives zero correct answers. The run-to-run difference is 0.0000 at the baseline and 0.0417 at the narrow budget. Sixteen lessons built and equipped an agent: the loop was written, the place where it stops was settled, side-effect calls were put behind approval, tools were defined, executed, and limited in capability, memory was fitted into the context window, long-term records were written, compacted, and expired. One question was never asked: how do we know this agent works. What usually remains of an agent run is a single rate: how many tasks were answered correctly. The claim of this lesson is that the rate stacks **four separate states** on top of each other, and that from inside the stack you cannot read whether the agent found the answer **by doing the work or without doing it**. The same distinction was built in the Embeddings and Retrieval-Augmented Generation course on the right chunk and the right answer; the procedure stayed there and is not repeated here. What carries over is only **the distinction itself**, this time over **steps**. - **AQ1.** Subscriber records, the document set, five tools, twenty-four tasks, and the agent loop are **fictional** and are generated inside the lesson. No real model endpoint or tool server is **called**; tool selection comes from a single draw that depends on depth. - **AQ2.** The **baseline is a single-step call with no tools**; the simulator sees only the request text. - **AQ3.** **Sequence measure**: if the task's expected tool sequence appears in the step trace **in order and without errors**, the sequence is correct; an extra call in between does not break the sequence. - **AQ4.** **Answer measure**: if the result the loop leaves equals the correct answer exactly, the answer is correct. - **AQ5.** **Step efficiency** is steps per correct answer; the other units of cost are **tool calls and tokens**, and no duration is written. - **AQ6.** Every measure is printed over **two runs** (seeds 20260218 and 20260219). **A difference below the run-to-run difference counts as unmeasured.** #### A Fictional Environment, Five Tools, and the Agent Loop Five tools read a meter, compute consumption, apply a tariff, search documents, and write a record; the last one has a side effect. Each task is solved by a known sequence. ```python # FICTIONAL. Subscriber records, document set, tool set, task set and agent # loop are generated within the lesson; no real endpoint or tool server is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "previous": previous, "current": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1) def t_read_meter(subscriber=None, **_): a = SUBSCRIBERS.get(subscriber) if a is None: return {"error": "no such subscriber"} return {"previous": a["previous"], "current": a["current"], "type": a["type"], "region": a["region"]} def t_consumption(previous=None, current=None, **_): if previous is None or current is None: return {"error": "missing field"} f = current - previous return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, type="residential", **_): if m3 is None: return {"error": "missing field"} return {"amount": _charge(m3, type)} def t_document(query=None, **_): if not query: return {"error": "missing field"} q = set(tokenize(query)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"} def t_record(subscriber=None, column=None, value=None, **_): if not (subscriber and column and value is not None): return {"error": "missing field"} return {"written": True, "subscriber": subscriber, "column": column, "value": value} TOOLS = { "read_meter": {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False}, "consumption": {"function": t_consumption, "fields": ("previous", "current"), "side_effect": False}, "tariff": {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False}, "search_docs": {"function": t_document, "fields": ("query",), "side_effect": False}, "write_record": {"function": t_record, "fields": ("subscriber", "column", "value"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"error": "no such tool"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("read_meter", "consumption", "tariff")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("read_meter", "consumption")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("search_docs",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("read_meter", "consumption", "tariff", "write_record")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["type"]) def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["approvals"] = state.get("approvals", 0) + 1 if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["result"] = s["m3"] if choice == "tariff": state["result"] = s["amount"] if choice == "search_docs": state["result"] = s["code"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} KINDS = ("charge", "consumption", "document", "record") counts = {t: sum(1 for g in TASKS if g["kind"] == t) for t in KINDS} print(f"fictional environment {len(SUBSCRIBERS)} subscribers , {len(DOCUMENTS)} documents , {len(TOOLS)} tools") print("tasks", len(TASKS), ",", " , ".join(f"{k} {v}" for k, v in counts.items())) ``` ``` fictional environment 60 subscribers , 5 documents , 5 tools tasks 24 , charge 6 , consumption 8 , document 7 , record 3 ``` #### Sequence and Answer: Two Separate Columns The first column never looks at the answer: did the expected sequence appear in the step trace in order and without errors. The second column never looks at the trace. Where the two columns cross, every task falls into one of four boxes. ```python def toolless(g, seed=SEED): """Baseline: a single-step call with no tools. No tool is called; the simulator sees only the request text and guesses from the code prefix.""" r = rng(seed ^ trace_id("tek" + g["name"])) prefix = {"sayac": "SY", "tarif": "TR", "ariza": "AR", "kesin": "KS"} b = set(tokenize(g["request"])) for k, p in prefix.items(): if k in b: candidates = [c for c in DOCUMENTS if c.startswith(p)] return candidates[int(r() * len(candidates))] return 10 * int(r() * 20) def sequence_correct(g, trace): """Does the expected tool sequence appear in the step trace in order and without errors.""" k = 0 for name, failed in trace: if k < len(g["sequence"]) and name == g["sequence"][k] and not failed: k += 1 return k == len(g["sequence"]) def four_states(tasks, budget, seed=SEED): """Budget zero runs the baseline, otherwise the loop runs.""" d = {"cc": 0, "cw": 0, "wc": 0, "ww": 0, "steps": 0, "tok": 0, "calls": 0} c0 = CALLS[0] for g in tasks: s = agent(g, max_steps=budget, seed=seed) if budget else None trace, answer, steps, tok = ((s["trace"], s["answer"], s["steps"], s["tokens"]) if s else ([], toolless(g, seed), 1, 60)) sc, ac = sequence_correct(g, trace), answer == correct_answer(g) d["cc" if sc and ac else "cw" if sc else "wc" if ac else "ww"] += 1 d["steps"] += steps d["tok"] += tok d["calls"] = CALLS[0] - c0 return d SETUPS = (("toolless", 0), ("budget 4", 4), ("budget 8", 8)) print("setup c+c c+w w+c w+w success 2nd run diff steps calls tokens steps/correct") for name, b in SETUPS: d, e = four_states(TASKS, b), four_states(TASKS, b, 20260219) n, m = (d["cc"] + d["wc"]) / 24, (e["cc"] + e["wc"]) / 24 print(f"{name:<9} {d['cc']:>3} {d['cw']:>4} {d['wc']:>4} {d['ww']:>4} {n:>8.4f} {m:>8.4f}" f" {m - n:>+9.4f} {d['steps']:>6} {d['calls']:>6} {d['tok']:>9}" f" {d['steps'] / max(n * 24, 1):>14.4f}") print("\ntask kind breakdown (c+c / c+w / w+c / w+w)") print(f"{'kind':<11} {'tasks':>6} {'toolless':>16} {'budget 8':>16}") for kind in KINDS: gs = [g for g in TASKS if g["kind"] == kind] row = [" / ".join(str(four_states(gs, b)[k]) for k in ("cc", "cw", "wc", "ww")) for b in (0, 8)] print(f"{kind:<11} {len(gs):>6} {row[0]:>16} {row[1]:>16}") needs_tools = [g for g in TASKS if g["kind"] != "document"] print(f"\n{len(needs_tools)} tasks that need tools , correct answers and steps") for name, b in SETUPS: d = four_states(needs_tools, b) print(f"{name:<9} correct {d['cc'] + d['wc']:>3} / {len(needs_tools)} steps {d['steps']:>4}") ``` ``` setup c+c c+w w+c w+w success 2nd run diff steps calls tokens steps/correct toolless 0 0 7 17 0.2917 0.2917 +0.0000 24 0 1440 3.4286 budget 4 20 0 1 3 0.8750 0.8333 -0.0417 84 64 9402 4.0000 budget 8 23 1 0 0 0.9583 0.9583 +0.0000 97 74 11978 4.2174 task kind breakdown (c+c / c+w / w+c / w+w) kind tasks toolless budget 8 charge 6 0 / 0 / 0 / 6 6 / 0 / 0 / 0 consumption 8 0 / 0 / 0 / 8 8 / 0 / 0 / 0 document 7 0 / 0 / 7 / 0 7 / 0 / 0 / 0 record 3 0 / 0 / 0 / 3 2 / 1 / 0 / 0 17 tasks that need tools , correct answers and steps toolless correct 0 / 17 steps 17 budget 4 correct 14 / 17 steps 64 budget 8 correct 16 / 17 steps 77 ``` #### Reading the Four Combinations The first row gives the baseline. One toolless step scores **0.2917**, and **all of that rate sits in the third box**: wrong sequence, right answer. The calls column is zero; all seven document tasks came out right, because the request of these tasks is the same text and the word sayac (meter) inside it gives away the prefix of the document code. One number says the agent solved more than a quarter of the tasks; the four-state table says on the same row that **no work was done**. The real demonstration is in the last row. At the eight-step budget the only failure is **in the second box**: right sequence, wrong answer. In one record task the four expected tools passed in order and without errors, but an extra document search was made between the tariff and the record, and that call replaced the result in the state with a document code. The agent did the right work in the right order, then spoiled its own result. The table writes here **which fix is needed**, because a tool selection defect and a state corruption defect are repaired in different places. The narrow budget row shows that the third box can fill in the loop as well. In one record task the right amount was found, but the budget did not reach the record write: sequence incomplete, answer right. A single number counts this task as a success, yet the side-effect step never happened. The breakdown by kind collects all seven right answers of the baseline under the document kind, and the only failure of the eight-step budget under the record kind. The two runs say which difference can be counted. At the narrow budget the difference is **0.0417**, a single task; at the baseline and at the eight-step budget it is **0.0000**. The rise of 0.0833 between the two budgets is twice the band and can be counted; the size of the individual box transitions **counts as unmeasured**. #### Steps per Correct Answer The second measure ties cost to success. At the eight-step budget the agent spends 97 steps and produces 23 correct answers: **4.2174** per step. At the narrow budget it is 84 steps and 21 correct answers, that is, **4.0000**. The trap is in the first row. The toolless baseline gives 7 correct answers in 24 steps, and step efficiency comes out at **3.4286**, the best of the three setups. The setup that calls no tool and computes nothing wins on the efficiency measure, because this measure puts only the correct answers in the denominator and never asks **which tasks could be solved**. The last table fixes the denominator. Counting only the seventeen tasks that need tools, the baseline gives **0 / 17** and efficiency stays undefined on this subset; the narrow budget gives 14 and the eight-step budget 16 correct answers. This is why the measure is not read alone: unless **the task set and the success level** it was measured at are written down, the shortest way to make an agent cheaper is to reduce the tasks it can solve. #### Summary - A single success rate stacks four states: right sequence with right answer, right sequence with wrong answer, wrong sequence with right answer, wrong sequence with wrong answer. - The toolless baseline comes out at 0.2917, and all of that rate sits in the **wrong sequence, right answer** box; the agent called no tool. - At the eight-step budget the success is 0.9583 and the only failure is in the **right sequence, wrong answer** box: the sequence was completed in full, and an extra call in between spoiled the result. - Steps per correct answer are 3.4286 for the baseline and 4.2174 at the eight-step budget; read alone, the measure rewards the baseline, because on the seventeen tasks that need tools the baseline gives zero correct answers. - The run-to-run difference is 0.0000 at the baseline and at the eight-step budget, and 0.0417 at the narrow budget; box transitions of a single task stay below this band. #### Next Step This lesson split the number of an agent run in two: whether the work was done and whether the answer is correct. Both assume one thing: that the tools themselves are correct. If one of the tools computed wrongly, swallowed a missing field silently, or had a definition that did not match its function, the table would still fill in and would not say which tool is broken. The next lesson separates every tool from the loop and tests them one by one: how many of the defects placed in the five tools does this testing catch, and **which defect class never shows up in this layer**. ### Testing Each Tool Separately from the Loop Source: https://academia.sh/en/courses/ai-agents/testing-each-tool-separately-from-the-loop Counting how many of eleven fictional defects placed in five tools the tool-level test catches: the eleven-case test suite fails on all four value defects and all three contract defects, but passes clean on all four definition defects. The overall catch rate is 0.6364, and by class 1.0000 and 1.0000 against 0.0000. The mechanism of the unseen class is written down: the test suite calls the tool's function directly, while the loop goes through the field list the tool declares, and a definition defect lives only in that list. The payoff of the layer is independence from the run: on the same task set the task the loop fails shifts from G23 to G15 when the seed changes, while the set of failing cases of the test suite is empty and identical in both runs. The previous lesson split an agent run into two columns: the correctness of the sequence and the correctness of the answer. Both assume the tools work correctly. The moment that assumption is lifted, the four-state table falls short: it says a task failed, it does not say which tool is broken. This lesson pulls the tools out of the loop and tests them one by one. The test levels were classified in the Quality and Testing Fundamentals course of the Software Quality and Testing curriculum, and the anatomy of the unit test was built in the Unit Testing and Test-Driven Development course; **the procedure is not repeated here**. The question belongs to measurement: **how many** of the defects placed in five tools show up in this layer, and **which class** this layer cannot see at all. - **AQ7.** The defects are **fictional** and are placed in a **copy** of the tool table; the original table does not change. - **AQ8.** There are three defect classes. **Value defect**: the function gives a wrong result. **Contract defect**: the function does not report an error where it should. **Definition defect**: the function is correct, the **declared field list** is wrong. - **AQ9.** **Tool-level test** tests a tool outside the loop, by calling its function directly; the expected output is written by hand. - **AQ10.** A defect counts as caught if it **fails at least one case** of the test suite. - **AQ11.** The test suite **uses no seed**; the loop measure is run with an eight-step budget and two seeds. #### Environment, Tool Definitions, and the Agent Loop A tool consists of two parts: the **function** does the computation, the **definition** declares which fields that function will take, and the loop builds the call only from the declared fields. ```python # FICTIONAL. Subscriber records, document set, tool set, task set and agent # loop are generated within the lesson; no real endpoint or tool server is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "previous": previous, "current": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1) def t_read_meter(subscriber=None, **_): a = SUBSCRIBERS.get(subscriber) if a is None: return {"error": "no such subscriber"} return {"previous": a["previous"], "current": a["current"], "type": a["type"], "region": a["region"]} def t_consumption(previous=None, current=None, **_): if previous is None or current is None: return {"error": "missing field"} f = current - previous return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, type="residential", **_): if m3 is None: return {"error": "missing field"} return {"amount": _charge(m3, type)} def t_document(query=None, **_): if not query: return {"error": "missing field"} q = set(tokenize(query)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"} def t_record(subscriber=None, column=None, value=None, **_): if not (subscriber and column and value is not None): return {"error": "missing field"} return {"written": True, "subscriber": subscriber, "column": column, "value": value} TOOLS = { "read_meter": {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False}, "consumption": {"function": t_consumption, "fields": ("previous", "current"), "side_effect": False}, "tariff": {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False}, "search_docs": {"function": t_document, "fields": ("query",), "side_effect": False}, "write_record": {"function": t_record, "fields": ("subscriber", "column", "value"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"error": "no such tool"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("read_meter", "consumption", "tariff")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("read_meter", "consumption")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("search_docs",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("read_meter", "consumption", "tariff", "write_record")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["type"]) def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["approvals"] = state.get("approvals", 0) + 1 if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["result"] = s["m3"] if choice == "tariff": state["result"] = s["amount"] if choice == "search_docs": state["result"] = s["code"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} print("tool declared fields side effect") for name, v in TOOLS.items(): print(f"{name:<12} {', '.join(v['fields']):<26} {v['side_effect']}") ``` ``` tool declared fields side effect read_meter subscriber False consumption previous, current False tariff m3, type False search_docs query False write_record subscriber, column, value True ``` #### Defect Set and Test Suite Eleven defects are spread over five tools. Value and contract defects change the **function**; definition defects leave the function alone and move only the **declared field list**. The test suite carries one ordinary call and one edge case for each tool. ```python DEFECTS = { # tool , class , description , defective function or field list "D1": ("read_meter", "value", "current index same as previous", lambda subscriber=None, **_: ({"error": "no such subscriber"} if subscriber not in SUBSCRIBERS else dict(SUBSCRIBERS[subscriber], current=SUBSCRIBERS[subscriber]["previous"]))), "D2": ("consumption", "value", "adds 10000 on meter rollover", lambda previous=None, current=None, **_: ( {"error": "missing field"} if previous is None or current is None else {"m3": current - previous + (10000 if current < previous else 0)})), "D3": ("tariff", "value", "upper tier unit 11 instead of 16", lambda m3=None, type="residential", **_: ( {"error": "missing field"} if m3 is None else {"amount": _charge(m3, type) - max(m3 - 40, 0) * 5 * (COMMERCIAL_FACTOR if type == "commercial" else 1)})), "D4": ("search_docs", "value", "first match instead of highest overlap", lambda query=None, **_: next( ({"code": k, "text": m} for k, m in DOCUMENTS.items() if set(tokenize(query or "")) & set(tokenize(m))), {"error": "missing field" if not query else "not found"})), "D5": ("consumption", "contract", "zero instead of error on missing field", lambda previous=None, current=None, **_: t_consumption(previous or 0, current or 0)), "D6": ("tariff", "contract", "zero instead of error on missing field", lambda m3=None, type="residential", **_: t_tariff(m3 or 0, type)), "D7": ("write_record", "contract", "says written when value is missing", lambda subscriber=None, column=None, value=None, **_: { "written": True, "subscriber": subscriber, "column": column, "value": value}), "D8": ("tariff", "definition", "definition does not declare the type field", ("m3",)), "D9": ("write_record", "definition", "definition does not declare the value field", ("subscriber", "column")), "D10": ("search_docs", "definition", "definition declares subscriber instead of query", ("subscriber",)), "D11": ("read_meter", "definition", "definition declares an unused field", ("subscriber", "column")), } CLASSES = ("value", "contract", "definition") def defective(code): """Copy of the table with a defect: if a tuple is given the declared field list, if a function is given the tool's function is replaced.""" t = {a: dict(v) for a, v in TOOLS.items()} name, defect = DEFECTS[code][0], DEFECTS[code][3] t[name]["fields" if isinstance(defect, tuple) else "function"] = defect return t A1 = "A001" TESTS = [ ("read_meter", {"subscriber": A1}, {"previous": SUBSCRIBERS[A1]["previous"], "current": SUBSCRIBERS[A1]["current"]}), ("read_meter", {"subscriber": "A999"}, {"error": "no such subscriber"}), ("consumption", {"previous": 100, "current": 150}, {"m3": 50}), ("consumption", {"previous": 99980, "current": 30}, {"m3": 50}), ("consumption", {"current": 150}, {"error": "missing field"}), ("tariff", {"m3": 50, "type": "commercial"}, {"amount": 940}), ("tariff", {}, {"error": "missing field"}), ("search_docs", {"query": "isyeri abonesinde birim ucret"}, {"code": "TR02"}), ("search_docs", {"query": ""}, {"error": "missing field"}), ("write_record", {"subscriber": A1, "column": "charge", "value": 500}, {"written": True}), ("write_record", {"subscriber": A1, "column": "charge"}, {"error": "missing field"}), ] def tool_test(table): """Tests each tool outside the loop, by calling its function directly.""" failed = [] for name, inp, expected in TESTS: c = table[name]["function"](**inp) if ("error" in c) != ("error" in expected) or any( c.get(k) != v for k, v in expected.items()): failed.append(name) return failed def loop_success(table, seed=SEED): global TOOLS original, TOOLS = TOOLS, table d = [g["name"] for g in TASKS if agent(g, max_steps=8, seed=seed)["answer"] != correct_answer(g)] TOOLS = original return d print(f"test suite {len(TESTS)} cases , failing on the defect-free table " f"{len(tool_test(TOOLS))}") print(f"defect set {len(DEFECTS)} defects , {len(TOOLS)} tools , {len(CLASSES)} classes\n") print(f"{'code':<5} {'tool':<12} {'class':<10} {'failed':>6} description") caught = {s: [0, 0] for s in CLASSES} for code, (name, cls, desc, defect) in DEFECTS.items(): d = tool_test(defective(code)) caught[cls][1] += 1 caught[cls][0] += 1 if d else 0 print(f"{code:<5} {name:<12} {cls:<10} {len(d):>6} {desc}") print("\nclass defects caught share") for s in CLASSES: a, b = caught[s] print(f"{s:<10} {b:>7} {a:>10} {a / b:>7.4f}") tp = sum(a for a, _ in caught.values()) print(f"{'total':<10} {len(DEFECTS):>7} {tp:>10} {tp / len(DEFECTS):>7.4f}") print("\ntwo runs , defect-free table") print(f"tool-level test , failing cases {tool_test(TOOLS)} and {tool_test(TOOLS)}") for t in (SEED, 20260219): d = loop_success(TOOLS, t) print(f"loop seed {t} , success {1 - len(d) / 24:.4f} , failing task {d}") ``` ``` test suite 11 cases , failing on the defect-free table 0 defect set 11 defects , 5 tools , 3 classes code tool class failed description D1 read_meter value 1 current index same as previous D2 consumption value 1 adds 10000 on meter rollover D3 tariff value 1 upper tier unit 11 instead of 16 D4 search_docs value 1 first match instead of highest overlap D5 consumption contract 1 zero instead of error on missing field D6 tariff contract 1 zero instead of error on missing field D7 write_record contract 1 says written when value is missing D8 tariff definition 0 definition does not declare the type field D9 write_record definition 0 definition does not declare the value field D10 search_docs definition 0 definition declares subscriber instead of query D11 read_meter definition 0 definition declares an unused field class defects caught share value 4 4 1.0000 contract 3 3 1.0000 definition 4 0 0.0000 total 11 7 0.6364 two runs , defect-free table tool-level test , failing cases [] and [] loop seed 20260218 , success 0.9583 , failing task ['G23'] loop seed 20260219 , success 0.9583 , failing task ['G15'] ``` #### Seven Caught, Four Unseen The strength of the layer is complete in the first two classes: all four value defects and all three contract defects fail a case in the test suite. That the number of failing cases is **one** each is also information; the defect shows in a single case written for its own tool, and the name of the failing case gives the broken tool directly. What the four-state table cannot say is said here in one line. The unseen class is not chance; it is the result of the call path. The test suite calls the tool's **function directly** and never looks at the declared field list. The loop, on the other hand, builds the call through `call_tool` and filters from the state only the **declared fields**. A definition defect sits exactly inside this filter: the `tariff` function applies the commercial factor correctly, but because its definition does not declare the `type` field, that field never passes in the loop and every commercial subscriber is computed from the residential tariff. A test that calls the function cannot see this, because the function is correct. For all four definition defects the result is **0.0000**. Total catch is **0.6364**. This number is specific to this defect set; what carries over is not the rate but **how the rate is divided**. A layer that tests the function is blind to the mismatch between function and definition, and this blindness does not close by adding cases. #### A Layer Independent of the Run The payoff of the layer is in the last block. On the defect-free tool table the loop gives **0.9583** at both seeds, but **the failing task changes**: `G23` in the first run, `G15` in the second. The tool-level test was run twice on the same table, and the set of failing cases came out **empty and identical** both times, because this layer uses no seed. The verdict of the tool-level test is **repeatable**: a defect either shows in every run or never shows. The verdict of the loop measure is **banded**: the difference between the two runs is a single task, and no sign below the band counts as a finding. #### Summary - A tool is the combination of a function and a definition; the tool-level test calls the function directly and never uses the definition. - The eleven-case test suite catches all four value defects and all three contract defects; each defect shows in a single case, and the failing case names the broken tool. - None of the four definition defects is caught: the share is **0.0000**, the total catch is **0.6364**. - The cause of the blindness is the call path: the loop filters only the declared fields, and the test suite calls the function directly and skips that filter. - The payoff of the layer is independence from the run: while the failing task of the loop shifts from G23 to G15 when the seed changes, the set of failing cases of the test suite is empty in both runs. #### Next Step This lesson drew the limit of a layer with a number: the tool-level test says cheaply and exactly whether a tool keeps its own word, but it never sees **how the tool is connected to the loop**. The four defects that go unseen have to show up somewhere, because the agent writes a wrong charge to commercial subscribers. The next lesson runs scenarios end to end and asks two questions together: how many of the defects that the tool level missed does the flow test catch, and **which defect do both layers fail to see**. ### Scenario-Based End-to-End Validation Source: https://academia.sh/en/courses/ai-agents/scenario-based-end-to-end-validation Counting side by side the defect sets caught by two testing layers: the twenty-four-scenario end-to-end flow test catches three of the four definition defects that the tool-level test cannot see at all, and fails on five of the eleven defects in total. The tool-level layer catches seven, the intersection of the two layers is two, and the union is ten of eleven defects, that is 0.9091. The two separate causes of a miss are told apart by the defect activation count: D2, D4, and D11 never activate once during the loop; D5, D6, and D7 activate but the scenario result does not change. The residual defect D11 is a defect whose function is correct but whose definition declares a field it never uses, and no run can see it. On the defect-free baseline the failing scenario in the two runs is G23 and G15. The previous lesson drew the limit of tool-level testing: seven of eleven defects were caught, and none of the four definition defects showed. Not being seen does not make them harmless; the charge of commercial subscribers is computed wrongly. These defects come out only when the tools are **connected to each other**. This lesson builds the second layer. The end-to-end flow test runs not the tool but the **scenario**, and at the end checks both the sequence and the answer. What is measured is not the layer but **the relation between the defect sets of the two layers**. - **AQ12.** The defect set is **the same** as in the previous lesson; the defects are placed in a copy of the table. **The tool-level column is the measurement of that lesson**; the test suite is not repeated here. - **AQ13.** **End-to-end flow test**: twenty-four scenarios are run with an eight-step budget. A scenario passes only if **both the sequence and the answer are correct**. - **AQ14.** On the defect-free table its baseline is **23 of 24**. A defect counts as caught by the flow layer if its failing scenario set differs from this baseline **in both runs**. - **AQ15.** **Defect activation count**: at every call the same input is also given to the defect-free table, and the counter goes up when the two results differ. **Zero activations** means the defect did not show up in the loop even once. - **AQ16.** The seeds are 20260218 and 20260219; a difference below the run-to-run difference **counts as unmeasured**. #### Fictional Environment and Agent Loop Every scenario has a known tool sequence and a known correct answer; the oracle of the test is this pair. ```python # FICTIONAL. Subscriber records, document set, tool set, task set and agent # loop are generated within the lesson; no real endpoint or tool server is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "previous": previous, "current": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1) def t_read_meter(subscriber=None, **_): a = SUBSCRIBERS.get(subscriber) if a is None: return {"error": "no such subscriber"} return {"previous": a["previous"], "current": a["current"], "type": a["type"], "region": a["region"]} def t_consumption(previous=None, current=None, **_): if previous is None or current is None: return {"error": "missing field"} f = current - previous return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, type="residential", **_): if m3 is None: return {"error": "missing field"} return {"amount": _charge(m3, type)} def t_document(query=None, **_): if not query: return {"error": "missing field"} q = set(tokenize(query)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"} def t_record(subscriber=None, column=None, value=None, **_): if not (subscriber and column and value is not None): return {"error": "missing field"} return {"written": True, "subscriber": subscriber, "column": column, "value": value} TOOLS = { "read_meter": {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False}, "consumption": {"function": t_consumption, "fields": ("previous", "current"), "side_effect": False}, "tariff": {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False}, "search_docs": {"function": t_document, "fields": ("query",), "side_effect": False}, "write_record": {"function": t_record, "fields": ("subscriber", "column", "value"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"error": "no such tool"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("read_meter", "consumption", "tariff")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("read_meter", "consumption")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("search_docs",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("read_meter", "consumption", "tariff", "write_record")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["type"]) def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["approvals"] = state.get("approvals", 0) + 1 if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["result"] = s["m3"] if choice == "tariff": state["result"] = s["amount"] if choice == "search_docs": state["result"] = s["code"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} print(f"fictional environment {len(SUBSCRIBERS)} subscribers , {len(TOOLS)} tools , " f"{len(TASKS)} scenarios") ``` ``` fictional environment 60 subscribers , 5 tools , 24 scenarios ``` #### Two Layers Side by Side The only thing the flow test sees is the sequence the loop produced and the answer. The defect activation counter puts one more column next to it: in how many calls the defect departed from the defect-free table in that run. The tool-level column comes from the previous lesson as input. ```python DEFECTS = { "D1": ("read_meter", "value", lambda subscriber=None, **_: ({"error": "no such subscriber"} if subscriber not in SUBSCRIBERS else dict(SUBSCRIBERS[subscriber], current=SUBSCRIBERS[subscriber]["previous"]))), "D2": ("consumption", "value", lambda previous=None, current=None, **_: ( {"error": "missing field"} if previous is None or current is None else {"m3": current - previous + (10000 if current < previous else 0)})), "D3": ("tariff", "value", lambda m3=None, type="residential", **_: ( {"error": "missing field"} if m3 is None else {"amount": _charge(m3, type) - max(m3 - 40, 0) * 5 * (COMMERCIAL_FACTOR if type == "commercial" else 1)})), "D4": ("search_docs", "value", lambda query=None, **_: next( ({"code": k, "text": m} for k, m in DOCUMENTS.items() if set(tokenize(query or "")) & set(tokenize(m))), {"error": "missing field" if not query else "not found"})), "D5": ("consumption", "contract", lambda previous=None, current=None, **_: t_consumption(previous or 0, current or 0)), "D6": ("tariff", "contract", lambda m3=None, type="residential", **_: t_tariff(m3 or 0, type)), "D7": ("write_record", "contract", lambda subscriber=None, column=None, value=None, **_: { "written": True, "subscriber": subscriber, "column": column, "value": value}), "D8": ("tariff", "definition", ("m3",)), # does not declare the type field "D9": ("write_record", "definition", ("subscriber", "column")), # does not declare the value field "D10": ("search_docs", "definition", ("subscriber",)), # subscriber instead of query "D11": ("read_meter", "definition", ("subscriber", "column")), # field the function does not use } CLEAN = {a: dict(v) for a, v in TOOLS.items()} def defective(code): """Copy of the table with a defect: if a tuple is given the declared field list, if a function is given the tool's function is replaced.""" t = {a: dict(v) for a, v in TOOLS.items()} name, p = DEFECTS[code][0], DEFECTS[code][2] t[name]["fields" if isinstance(p, tuple) else "function"] = p return t # The layer of the previous lesson is input: the eleven-case test suite was built there # and these are the defects it failed. The computation is not repeated here. TOOL_LEVEL = {"D1", "D2", "D3", "D4", "D5", "D6", "D7"} ACTIVATIONS = [0] _call_tool = call_tool def call_tool(name, state): """Builds the call with the table in force, also gives the same input to the defect-free table, and counts an activation when the two results differ.""" b = _call_tool(name, state) kw = {a: state[a] for a in CLEAN[name]["fields"] if a in state} ACTIVATIONS[0] += b != CLEAN[name]["function"](**kw) return b def sequence_correct(g, trace): k = 0 for name, failed in trace: if k < len(g["sequence"]) and name == g["sequence"][k] and not failed: k += 1 return k == len(g["sequence"]) def flow_test(table, seed=SEED): """End-to-end scenario: it passes if both the sequence and the answer are correct.""" global TOOLS original, TOOLS, ACTIVATIONS[0] = TOOLS, table, 0 d = [] for g in TASKS: s = agent(g, max_steps=8, seed=seed) if not (sequence_correct(g, s["trace"]) and s["answer"] == correct_answer(g)): d.append(g["name"]) TOOLS = original return d, ACTIVATIONS[0] baseline = [flow_test(CLEAN, t) for t in (SEED, 20260219)] print(f"flow test {len(TASKS)} scenarios , failing scenario on the defect-free table " f"{baseline[0][0]} and {baseline[1][0]}") print("\ncode tool class tool-level flow-1 flow-2 act-1 act-2") by_tool, by_flow = set(), set() for code, (name, cls, p) in DEFECTS.items(): t, at_tool = defective(code), code in TOOL_LEVEL (d1, a1), (d2, a2) = flow_test(t, SEED), flow_test(t, 20260219) if at_tool: by_tool.add(code) if d1 != baseline[0][0] and d2 != baseline[1][0]: by_flow.add(code) print(f"{code:<5} {name:<12} {cls:<10} {'+' if at_tool else '-':>10} {len(d1):>7} " f"{len(d2):>7} {a1:>7} {a2:>7}") n = len(DEFECTS) print() for label, s in (("both", by_tool & by_flow), ("flow only", by_flow - by_tool), ("tool level only", by_tool - by_flow), ("residual", set(DEFECTS) - by_tool - by_flow)): print(f"{label:<16} {' '.join(k for k in DEFECTS if k in s)}") print(f"tool level {len(by_tool)}/{n} = {len(by_tool) / n:.4f} , flow {len(by_flow)}/{n} = " f"{len(by_flow) / n:.4f} , union {len(by_tool | by_flow)}/{n} = " f"{len(by_tool | by_flow) / n:.4f}") ``` ``` flow test 24 scenarios , failing scenario on the defect-free table ['G23'] and ['G15'] code tool class tool-level flow-1 flow-2 act-1 act-2 D1 read_meter value + 17 17 18 20 D2 consumption value + 1 1 0 0 D3 tariff value + 7 7 9 8 D4 search_docs value + 1 1 0 0 D5 consumption contract + 1 1 2 1 D6 tariff contract + 1 1 4 5 D7 write_record contract + 1 1 2 1 D8 tariff definition - 5 6 6 5 D9 write_record definition - 3 3 7 14 D10 search_docs definition - 7 8 44 49 D11 read_meter definition - 1 1 0 0 both D1 D3 flow only D8 D9 D10 tool level only D2 D4 D5 D6 D7 residual D11 tool level 7/11 = 0.6364 , flow 5/11 = 0.4545 , union 10/11 = 0.9091 ``` #### Two Sets That Do Not Intersect The tool-level layer catches **7/11**, the flow layer **5/11**. The intersection is only **two defects**: `D1` and `D3`. `D1` activates on every meter reading and fails **seventeen** of the twenty-four scenarios; `D3` activates only on consumptions that exceed the upper tier, and fails **seven** of them. The real information is outside the intersection. The flow layer catches **`D8`, `D9`, `D10`**, that is, three of the four definition defects the tool level could not see at all. The reason is in the mechanism: the flow test sees the tool not through its function but through **the call the loop builds**, and that call passes through the declared field list. `D10` gives the highest activation count: **44** and **49**. In the opposite direction **`D2`, `D4`, `D5`, `D6`, `D7`** show up only at the tool level. The union is **10/11 = 0.9091**; neither layer alone comes close to it. #### Defects That Do Not Activate, and the Residual The defects the flow misses have **two separate causes**, and the activation column tells them apart. `D2` and `D4` activate **zero** times in both runs: the meter rollover branch is never entered in this task set, and in the document query the first overlapping document is already the one with the highest overlap. The defect did not show up even once; the flow had no chance of finding it. `D5`, `D6`, and `D7`, by contrast, do activate, in 2, 4, and 2 calls respectively, but the failing scenario set does not change: the defect was run, it spoiled the result, and **the scenario passed anyway**. The first is a gap in coverage, the second a gap in observation. The activation count is not read as a magnitude. `D9` activates 7 times in the first run and 14 in the second; `D1` 18 and 20. What is read is **whether it is zero**. The residual defect is a single defect and its name is `D11`. The `read_meter` function is correct, but its definition declares a field the function never uses. The tool-level test cannot see it, because the function works correctly. The flow test cannot see it, because the extra declared field enters the call and is swallowed silently: the activation count is **zero** in both runs. The defect is not in the behavior but **in the contract itself**, and the one who will read it is someone else, who will call the tool. #### Summary - The flow test runs the scenario and sees only the sequence and the answer; the failing scenario on the defect-free baseline is G23 and G15 in the two runs. - The flow layer catches 5 of the 11 defects, the tool-level layer 7; the intersection is only `D1` and `D3`, and the union is 10/11 = **0.9091**. - The flow layer catches three of the four definition defects the tool level could not see, because it passes the call through the declared field list. - The cause of a miss splits in two: `D2`, `D4`, and `D11` never activate; `D5`, `D6`, and `D7` activate but do not change the scenario result. - The residual defect is `D11`: the function is correct, the definition declares an extra field; because the activation count is zero, no run can show this defect. #### Next Step Together the two layers found ten of the eleven defects, but they have one thing in common: we **put the defect in** and knew from the start where it was. Testing sees the runs in which the agent is tested; it says nothing about a loop running in production. There is no failing scenario there, only a user and a wrong answer. The next lesson measures what a run leaves behind: at three trace levels, the share of failed tasks whose cause can be found, and the stored token. ### Monitoring and Logging Source: https://academia.sh/en/courses/ai-agents/monitoring-and-logging Measuring what the step trace of an agent loop running in production reads back: in a 240-run stream made of ten runs of twenty-four tasks, fifteen tasks end wrong, and their causes are twelve wrong tool selections and three cases of wandering. The trace that stores only the result gives 0.0000 of these causes, the trace that counts tool names gives 0.4000, and the trace that stores the full observation gives 1.0000; the stored token rises from 2738 to 5954 and 13623, while the token per cause found falls from 992.3 to 908.2. 0.8689 of the sixty-one tool errors occur in runs that end correctly, that is, the tool error counter is a symptom, not a cause. The previous two lessons tested the agent. Tool-level testing probed, one by one, the defects placed in five tools; the end-to-end flow test caught what it missed, and a residual remained that neither could see. Both share one constraint: **we built the run being tested**. We knew in advance which task was running, which tool was broken, and what the correct answer was. In a loop running in production, none of these is at hand. The task comes from a subscriber, the agent chooses its own steps, the result is written to a record, and what remains is **only what we stored**. What this lesson measures is not the agent itself but **what the trace reads back**: when a task ends wrong, whether the record we stored can say **why** it ended wrong. The general procedure of observability was set up in the Observability and Reliability course and in the Observability and Operations course; it is not repeated here. The only thing measured is the trade-off between **the information the step trace carries and its cost**. - **AQ31.** The environment, tools, task set, and agent loop are **fictional** and are defined within the lesson below; no real model endpoint, tool server, or runtime is called. The seed is **20260218**, the second run **20260219**. - **AQ32. The production stream** is the same twenty-four tasks run **ten times**. Because the model is a nondeterministic component, the agent's draw is separate in each run; the total is **240 runs**. The step budget is **6**, that is, the value that came out best in the sweep of `agent-architecture/04`. - **AQ33. Three trace levels** store the following. **Result only**: the task name, the final answer, and whether the loop stopped. **Tool names**: in addition to these, how many times each tool was called — **the order is not stored**, because this level is a counter. **Full observation**: in addition to these, **the order of the calls and the whole of the observation each call returned**. - **AQ34. The cause** is the **first departure** in the trace and is read from the full observation: if a tool outside the plan was called before the plan finished, **wrong tool selection**; if calls continued after the plan finished, **wandering**; if a tool in the plan returned an error in its turn, **tool error**; if the plan did not finish with no departure at all, **budget**. - **AQ35.** A trace level gives a cause only **if the fields it stores determine it**; where it stays ambiguous, the cause counts as **not found**, and a wrong guess is not counted as right. - **AQ36.** The stored token is counted by passing the record through the shared tokenizer. The term used in this course for the trace is **step trace**. #### Production Stream and Shared Definition The block below sets up the shared definition of the course: the subscriber records of the measurement network, the fictional document set, five tools, twenty-four tasks, and the agent loop. Tool selection degrades with depth, the agent stops with a share of 0.88 when the work is done, fields outside the schema are filtered out, and a missing field is the tool's own error. The tool and field names in the code are the `read_meter`, `consumption`, `tariff`, `search_docs`, and `write_record` tools of the earlier lessons; they are counted in tokens here, so they stay Turkish. ```python # FICTIONAL. The measurement network's subscriber records, document set, five tools and # twenty-four tasks are defined within the lesson; no real endpoint or tool server is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # wire words stay Turkish: they are counted in tokens (same rule as the MCP course) # onceki/simdiki/tip/bolge = previous/current/type/region; mesken/isyeri = residential/commercial REGION = ["dogu", "bati", "kuzey", "guney", "merkez"] TYPE = ["mesken", "isyeri"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"bolge": REGION[int(r() * 5)], "tip": TYPE[int(r() * 2)], "onceki": previous, "simdiki": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "isyeri" else 1) def t_read_meter(abone=None, **_): a = SUBSCRIBERS.get(abone) if a is None: return {"hata": "abone yok"} return {"onceki": a["onceki"], "simdiki": a["simdiki"], "tip": a["tip"], "bolge": a["bolge"]} def t_consumption(onceki=None, simdiki=None, **_): if onceki is None or simdiki is None: return {"hata": "eksik alan"} f = simdiki - onceki return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, tip="mesken", **_): if m3 is None: return {"hata": "eksik alan"} return {"tutar": _charge(m3, tip)} def t_document(anahtar=None, **_): if not anahtar: return {"hata": "eksik alan"} q = set(tokenize(anahtar)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"kod": code, "metin": DOCUMENTS[code]} if best > 0 else {"hata": "bulunamadi"} def t_record(abone=None, sutun=None, deger=None, **_): if not (abone and sutun and deger is not None): return {"hata": "eksik alan"} return {"yazildi": True, "abone": abone, "sutun": sutun, "deger": deger} TOOLS = { "sayac_oku": {"function": t_read_meter, "fields": ("abone",), "side_effect": False}, "tuketim": {"function": t_consumption, "fields": ("onceki", "simdiki"), "side_effect": False}, "tarife": {"function": t_tariff, "fields": ("m3", "tip"), "side_effect": False}, "belge_ara": {"function": t_document, "fields": ("anahtar",), "side_effect": False}, "kayit_yaz": {"function": t_record, "fields": ("abone", "sutun", "deger"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"hata": "arac yok"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("sayac_oku", "tuketim", "tarife")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("sayac_oku", "tuketim")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("belge_ara",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("sayac_oku", "tuketim", "tarife", "kayit_yaz")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["simdiki"] - a["onceki"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["tip"]) def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"abone": g["subscriber"], "anahtar": g["request"], "sutun": "bedel"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("sonuc"), "approvals": state.get("onay", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["onay"] = state.get("onay", 0) + 1 if choice == "kayit_yaz": state["deger"] = state.get("sonuc") s = call_tool(choice, state) trace.append((choice, "hata" in s)) tok += 30 if "hata" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "hata"}) if choice == "tuketim": state["sonuc"] = s["m3"] if choice == "tarife": state["sonuc"] = s["tutar"] if choice == "belge_ara": state["sonuc"] = s["kod"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("sonuc"), "approvals": state.get("onay", 0)} ``` #### What the Trace Reads Back The same flow runs for each of the three levels, the records are built, and the cause of the failed tasks is derived from the fields that level stored. To build the full observation level, the tool call is wrapped: the whole of the returned dictionary lands in the log. ```python # Production stream: the same task set runs ten times, and the agent's draw is separate in each run. BUDGET, RUNS = 6, 10 OBS_LOG = [] _raw_call = call_tool def call_tool(name, state): # full observation level: the returned fields are stored too s = _raw_call(name, state) OBS_LOG.append((name, dict(s))) return s def cause(g, trace): """Cause read from the full observation: the first departure in the trace.""" k = 0 for name, obs in trace: if k >= len(g["sequence"]): return "wandering" if name != g["sequence"][k]: return "wrong tool selection" if "hata" in obs: return "tool error" k += 1 return "budget" if k < len(g["sequence"]) else "unknown" def diagnose_names(g, counts): """The name level keeps only a counter: it has neither the order nor the observation.""" if not [a for a in counts if a not in g["sequence"]]: return None if [a for a in g["sequence"] if counts.get(a, 0) == 0]: return "wrong tool selection" if all(counts[a] == g["sequence"].count(a) for a in set(g["sequence"])): return "wandering" return None summary = {} for base_seed in (20260218, 20260219): found, stored, failed, distribution, errors, errors_correct = [0, 0, 0], [0, 0, 0], 0, {}, 0, 0 for d in range(RUNS): for g in TASKS: OBS_LOG.clear() s = agent(g, max_steps=BUDGET, seed=base_seed + 7 * d) trace = list(OBS_LOG) counts = {} for a, _ in trace: counts[a] = counts.get(a, 0) + 1 base_record = {"ad": g["name"], "yanit": s["answer"], "bitti": s["done"]} record = (base_record, dict(base_record, sayac=counts), dict(base_record, iz=trace)) for i in range(3): stored[i] += len(tokenize(record[i])) errors += s["errors"] if s["answer"] == correct_answer(g): errors_correct += s["errors"] continue failed += 1 actual = cause(g, trace) distribution[actual] = distribution.get(actual, 0) + 1 found[1] += diagnose_names(g, counts) == actual found[2] += cause(g, trace) == actual summary[base_seed] = (found, stored, failed) breakdown = " , ".join(f"{k} {v}" for k, v in sorted(distribution.items())) print(f"seed {base_seed} , {RUNS * len(TASKS)} runs , failed {failed} , cause: {breakdown}") print(f" tool errors {errors} , their share in runs that end correct {errors_correct / errors:.4f}") print("\ntrace level cause found share stored token per cause found") for i, name in enumerate(("result only", "tool names", "full observation")): a, b = summary[20260218], summary[20260219] share = f"{a[0][i] / a[2]:.4f} {b[0][i] / b[2]:.4f}" tok = f"{a[1][i]:>6} {b[1][i]:>6}" per = " ".join(f"{x[1][i] / x[0][i]:>7.1f}" if x[0][i] else " -" for x in (a, b)) print(f"{name:<18} {share} {tok} {per}") ``` ``` seed 20260218 , 240 runs , failed 15 , cause: wandering 3 , wrong tool selection 12 tool errors 61 , their share in runs that end correct 0.8689 seed 20260219 , 240 runs , failed 11 , cause: wandering 3 , wrong tool selection 8 tool errors 66 , their share in runs that end correct 0.9242 trace level cause found share stored token per cause found result only 0.0000 0.0000 2738 2739 - - tool names 0.4000 0.2727 5954 6043 992.3 2014.3 full observation 1.0000 1.0000 13623 14486 908.2 1316.9 ``` A trace that stores only the result gives **none** of the causes. This level can **count** failure: it knows that fifteen tasks ended wrong. When asked for the cause, all it has is the answer and the stop flag, and these two fields do not tell apart two separate paths that produce the same wrong answer. A dashboard can be built with this level; a fault cannot be closed with it. The trace that counts tool names gives **0.4000** of the causes. What this level knows is which tool was called how many times; there is no order. If an unplanned tool appears in the counter and **none** of the planned tools has been called, the plan is incomplete, that is, the departure is a wrong tool selection. If there is an unplanned tool but each of the planned tools has been called exactly once, the plan is finished and the extra call came afterward, that is, it is wandering. Outside these two cases the counter **cannot tell** two scenarios apart: the same counter can come from a wrong selection made in the middle of the plan and also from wandering after the plan finished. In those runs the cause counts as not found. The full observation level gives **all** of the causes, because the order is stored together with the observation each call returned, and the plan pointer can be replayed exactly. Its cost is that the stored token rises from 2738 to **13623** — five times. The last column changes the ranking. When read as **token per cause found**, the tool names level is 992.3 and the full observation level is **908.2**. The level that is five times more expensive in total is **cheaper** per unit of useful output, because most of the token the cheap level stores closes no question. In the second run the same reading is sharper still: 2014.3 against 1316.9. #### A Tool Error Is a Symptom, Not a Cause There are **61 tool errors** in the stream, and **0.8689 of them occur in runs that end correctly**. An alert placed on the tool error counter fires sixty-one times, and fifty-three of those are for a task that ended without trouble. It is deficient in the opposite direction too: the **tool error class never appears** in the table of causes. The mechanism explains this. When a tool in the plan has its turn, its input is already ready in the state and it returns no error; an error comes only when a **wrongly selected** tool cannot find the field its schema asks for. So a tool error always comes **after** a wrong selection, and looking at the error count without correcting that selection leads one to fix the tool. The two runs also draw the limit of this reading. The tool names level gives 0.4000 in one run and 0.2727 in the other; the difference of **0.1273** between them is the measurement band. That this level is **0.6000** behind the full observation is large relative to the band and is a finding. But how much this level's own value moved between the two runs says that a claim of the form "tool names give this much of the causes" is **unmeasured on its own**; what is measured is the order among the levels. #### Summary - The production stream is ten runs of twenty-four tasks; 15 of the 240 runs end wrong, and their causes are 12 wrong tool selections and 3 cases of wandering. - The result only level gives 0.0000 of the causes, the tool names level 0.4000, and the full observation level 1.0000. - The cost is 2738, 5954, and 13623 tokens; read per cause found, the ranking reverses and the full observation level drops to 908.2, below the 992.3 of tool names. - 0.8689 of the 61 tool errors are in runs that end correctly, and the tool error class never appears in the table of causes: the error counter is a symptom, not a cause. - The difference of the tool names level between the two runs is 0.1273; the value of this level on its own counts as unmeasured, and the order among the levels counts as measured. #### Next Step The trace reads back why a task ended wrong, and this has a token cost. But this cost is only one item of the bill the loop runs up. The agent makes one call at each step, every call spends tokens, and the number of steps changes from task to task. The next lesson reads this spending with two separate denominators: **per step** and **per correct answer**. There is a point where the two rankings are not the same, and at that point which setup counts as cheap depends on the choice of denominator. ### Cost and Latency Control Source: https://academia.sh/en/courses/ai-agents/cost-and-latency-control Reading the spending of an agent loop with two separate denominators: the single-step call without tools is the cheapest at 90.00 cost units per step, but because its success is 0.0000, its cost per correct answer is undefined. When the large and small model classes are run with three step budgets, the small class stays below the 118.30 of the large class at 79.25 per step, while at 488.0 per correct answer it rises above its 453.5. The band where the two rankings diverge is given by the unit price ratio: the break-even ratio is 0.8957 per step and 0.5576 per correct answer, and at every price ratio in between the two measures give opposite answers. Latency is written as a class and read from the longest run; raw time is not measured. The previous lesson paid for the trace in tokens: the full observation level stored 13623 tokens and, in return, read back the cause of the failed tasks. That cost is only one item of the spending the loop runs up. The real spending is in the loop itself: each step carries a prompt, the carried prompt grows with the previous observations, and the number of steps changes from task to task. The question of this lesson is this: **what do we divide by** when we say a setup is expensive or not. The cost lesson in the Building Applications with Model APIs course set up token accounting per request and showed there too that the ranking changes when the denominator changes; that procedure is not repeated here. Here the denominator is **the step** or **the correct answer**, the thing divided is the spending not of a single call but of **the whole loop**, and the things compared are the step budget and the model class. - **AQ41.** The environment, tools, task set, and agent loop are **fictional**; no real model endpoint or tool server is called. The seed is **20260218**, the second run **20260219**. - **AQ42. The baseline is the single-step call without tools**: the agent calls no tool and answers in a single step. Its token is the token of the first step of the loop (prompt 60, answer 30). Because no observation enters the state, its success is **0.0000**. - **AQ43. The model class** scales the share of selecting the right tool. The multiplier of the **large class** is 1.00, that is, the shared definition itself; the multiplier of the **small class** is **0.60**, and the share is clipped between 0.20 and 0.95. The classes are referred to by type name; no model, version, or provider name appears. - **AQ44. The cost unit** is the relative price paid per token. It is taken as 1.00 in the large class; the unit price of the small class is a **parameter** and is printed in the table as **0.60**. - **AQ45. Latency is written as a class and is read from the step count of the longest run**: 1 step **single call**, 2 to 4 steps **short chain**, 5 to 8 steps **long chain**, nine and above **very long chain**. Raw time is not measured anywhere; what determines the class is the length of the chain in the worst case. - **AQ46. The break-even ratio** is the value of the unit price of the small class at which the two classes tie on the measure in question. It is derived independently of the chosen price and is read from the table itself. #### Six Setups, One Task Set The block below sets up the shared definition of the course: the fictional subscriber records of the measurement network, the fictional document set, five tools, twenty-four tasks, and the agent loop. ```python # FICTIONAL. The measurement network's subscriber records, document set, five tools and # twenty-four tasks are defined within the lesson; no real endpoint or tool server is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "previous": previous, "current": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1) def t_read_meter(subscriber=None, **_): a = SUBSCRIBERS.get(subscriber) if a is None: return {"error": "no such subscriber"} return {"previous": a["previous"], "current": a["current"], "type": a["type"], "region": a["region"]} def t_consumption(previous=None, current=None, **_): if previous is None or current is None: return {"error": "missing field"} f = current - previous return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, type="residential", **_): if m3 is None: return {"error": "missing field"} return {"amount": _charge(m3, type)} def t_document(query=None, **_): if not query: return {"error": "missing field"} q = set(tokenize(query)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"} def t_record(subscriber=None, column=None, value=None, **_): if not (subscriber and column and value is not None): return {"error": "missing field"} return {"written": True, "subscriber": subscriber, "column": column, "value": value} TOOLS = { "read_meter": {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False}, "consumption": {"function": t_consumption, "fields": ("previous", "current"), "side_effect": False}, "tariff": {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False}, "search_docs": {"function": t_document, "fields": ("query",), "side_effect": False}, "write_record": {"function": t_record, "fields": ("subscriber", "column", "value"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"error": "no such tool"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("read_meter", "consumption", "tariff")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("read_meter", "consumption")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("search_docs",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("read_meter", "consumption", "tariff", "write_record")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["type"]) def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["approvals"] = state.get("approvals", 0) + 1 if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["result"] = s["m3"] if choice == "tariff": state["result"] = s["amount"] if choice == "search_docs": state["result"] = s["code"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} ``` The model class touches the shared definition at a single place: a multiplier that scales the share of selecting the right tool. With the multiplier at 1.00 the baseline is kept exactly, that is, the large class is the shared definition itself. ```python # The model class scales the share of selecting the right tool; the large class is the shared definition itself. MULTIPLIER = [1.00] _base_selection = selection def selection(d): return max(0.20, min(0.95, _base_selection(d) * MULTIPLIER[0])) MODEL_CLASS = {"large": 1.00, "small": 0.60} # multiplier of the share of selecting the right tool UNIT_PRICE = {"large": 1.00, "small": 0.60} # cost unit per token NO_TOOL_TOKENS = 60 + 30 # token of the single-step call without tools def latency(longest): if longest <= 1: return "single call" return "short chain" if longest <= 4 else ("long chain" if longest <= 8 else "very long chain") def run(cls, budget, seed): MULTIPLIER[0] = MODEL_CLASS[cls] correct, steps, tok, longest = 0, 0, 0, 0 for g in TASKS: s = agent(g, max_steps=budget, seed=seed) steps += s["steps"] tok += s["tokens"] longest = max(longest, s["steps"]) correct += s["answer"] == correct_answer(g) return correct, steps, tok, longest for seed in (20260218, 20260219): n = len(TASKS) print(f"seed {seed}") print("setup success steps tokens cost per step per correct latency") print(f"{'no tools':<15} {0.0:>8.4f} {n:>6} {n * NO_TOOL_TOKENS:>9} {n * NO_TOOL_TOKENS:>8.1f}" f" {NO_TOOL_TOKENS:>12.2f} {'undefined':>13} {latency(1)}") data = {} for cls in ("large", "small"): for budget in (4, 6, 12): correct, steps, tok, longest = run(cls, budget, seed) data[(cls, budget)] = (correct, steps, tok) cost = tok * UNIT_PRICE[cls] print(f"{cls + ' budget ' + str(budget):<15} {correct / n:>8.4f} {steps:>6} {tok:>9} {cost:>8.1f}" f" {cost / steps:>12.2f} {cost / correct:>13.1f} {latency(longest)}") print("budget break-even ratio per step per correct answer") for budget in (4, 6, 12): lc, ls, lt = data[("large", budget)] sc, ss, st = data[("small", budget)] print(f"{budget:>6} {(lt / ls) / (st / ss):>28.4f} {(lt / lc) / (st / sc):>20.4f}") ``` ``` seed 20260218 setup success steps tokens cost per step per correct latency no tools 0.0000 24 2160 2160.0 90.00 undefined single call large budget 4 0.8750 84 9402 9402.0 111.93 447.7 short chain large budget 6 1.0000 92 10884 10884.0 118.30 453.5 long chain large budget 12 0.9583 98 12214 12214.0 124.63 531.0 very long chain small budget 4 0.5417 90 10624 6374.4 70.83 490.3 short chain small budget 6 0.7917 117 15454 9272.4 79.25 488.0 long chain small budget 12 1.0000 142 21762 13057.2 91.95 544.0 very long chain budget break-even ratio per step per correct answer 4 0.9482 0.5478 6 0.8957 0.5576 12 0.8132 0.5857 seed 20260219 setup success steps tokens cost per step per correct latency no tools 0.0000 24 2160 2160.0 90.00 undefined single call large budget 4 0.8333 83 9366 9366.0 112.84 468.3 short chain large budget 6 0.9583 96 11610 11610.0 120.94 504.8 long chain large budget 12 0.9583 98 11994 11994.0 122.39 521.5 long chain small budget 4 0.3750 89 10550 6330.0 71.12 703.3 short chain small budget 6 0.7917 121 16396 9837.6 81.30 517.8 long chain small budget 12 0.8750 140 20526 12315.6 87.97 586.5 very long chain budget break-even ratio per step per correct answer 4 0.9519 0.3995 6 0.8925 0.5850 12 0.8348 0.5335 ``` #### Cheap Relative to What The first row sets up the question by itself. The call without tools is cheaper than every setup of the large class at **90.00 per step**; it is below 111.93, 118.30, and 124.63. Its cost per correct answer is **undefined**, because the denominator is zero. At the very top of the cost-per-step ranking sits the setup that finishes no work. This is not an error of the measure but the result of what the denominator counts: cost per step measures **the cheapness of the step**; it does not measure whether the work got done. Within the same class, enlarging the budget steadily raises the cost per step: 111.93, 118.30, 124.63 in the large class. The reason is in the loop's token accounting; each step also carries the observations before it, so late steps are more expensive than early ones. The cost per correct answer, by contrast, is 447.7, 453.5, and 531.0. The difference of 5.8 between the first two values is smaller than the difference between the two runs of the same row (447.7 against 468.3, that is, 20.6); the ranking between budget 4 and budget 6 **counts as unmeasured** on this measure. The 531.0 of budget 12 is large relative to the band and is a real deterioration. #### The Band Where the Two Rankings Part The main divergence is between the classes. With the unit price taken as 0.60, the small class at budget 6 is clearly below the large class's **118.30** per step, at **79.25**. Read per correct answer, the same two rows reverse their order: **488.0** against **453.5**. The small class produces cheap steps but spends more of them (117 against 92) and finishes fewer tasks (0.7917 against 1.0000). The second run gives the same direction: 81.30 against 120.94 and 517.8 against 504.8. This divergence may look dependent on the chosen price, but that it is not can be read from the last section of the table. At budget 6 the break-even ratio per step is **0.8957** and the break-even ratio per correct answer is **0.5576**. If the unit price of the small class is below 0.5576, both measures count the small class as cheap; if it is above 0.8957, both measures count the large class as cheap. **At every price ratio in between, the two measures give opposite answers**, and the question of which one is cheap has no single right answer. The second run puts the band between 0.5850 and 0.8925; the upper edges of the two runs overlap by 0.0032, and the lower edge moves by 0.0274. For this reason the band is roughly between **0.56 and 0.89**. The lower edge at budget 4, however, moves between 0.5478 and 0.3995; the band of that row **counts as unmeasured**. The latency column carries a separate warning. The class is read from the longest run, and the large class at budget 12 comes out **very long chain** in one run and **long chain** in the other. The mean step count is almost the same in the two runs; what moves is the worst case of a single task. If a latency budget is set by the worst case, the class of the same setup can change between two runs, and this change cannot be read as an improvement or a deterioration. #### Summary - The single-step call without tools is the cheapest setup at 90.00 per step and its cost per correct answer is undefined; the per-step measure does not count whether the work got done. - In the large class, as the budget grows, the cost per step rises from 111.93 to 124.63, because late steps also carry the previous observations. - At a unit price of 0.60, the small class is cheap at 79.25 per step and expensive at 488.0 per correct answer; the counterparts of the large class are 118.30 and 453.5. - The break-even ratios give the band where the two measures part: at budget 6, 0.8957 per step and 0.5576 per correct answer; at every price in between, the two measures give opposite answers. - Latency is a class read from the longest run, and the same setup can fall into two different classes in two runs; raw time is not measured anywhere. #### Next Step One column stayed silent in all the rows of the cost table: the tool error. What the loop does when a tool returns an error was never asked in this lesson; the loop of the shared definition saw the error and turned toward the same goal again, and we had not written this down as a **policy**. The next lesson measures this gap: stopping at the first error, retrying the same tool, and never selecting the failed tool again are three separate policies, and the three pull the success, the steps, and the tool errors in separate directions. That lesson is also the close of the course. ### Error Recovery Source: https://academia.sh/en/courses/ai-agents/error-recovery Measuring what a failed tool call does to the loop with three recovery policies: in a production stream of two hundred forty runs, stopping at the first error gives a task success of 0.7500, 727 steps, 53 tool errors, and 81996 tokens; retrying the same tool raises success to 0.9375 and moves the steps to 900, the tool errors to 61, and the tokens to 106352. Dropping the failed tool and moving on to another does the opposite of what is expected: tool errors rise to 64, steps to 907, and success falls to 0.8917. The step efficiency that stopping gains stays within the band of the two runs. The lesson is the close of the AI Agents course and collects the without-scaffolding and with-scaffolding numbers of twenty-two lessons in a single table. One column stayed silent from start to finish in the cost table of the previous lesson: the tool error. Errors occurred in all six setups, and in none of them was it written what to do. When the loop of the shared definition sees an error, it turns toward the same target again; this is a behavior but not a policy, because it was not chosen. The retry, backoff, and circuit breaker procedure was measured in the Resilience and Reliability course; it is not repeated. The only thing measured is **what the failed tool does to the loop**. - **AQ51.** The environment, tools, tasks, and loop are **fictional**. The production stream is the same as that of the previous lesson's: twenty-four tasks run **ten times**, the budget is **6**, and the total is **240 runs**. The seeds are **20260218** and **20260219**. - **AQ52. Stop**: the loop ends at the first error; this is the baseline. **Retry**: the target does not change; at the next step the loop turns to the same target again. **Skip**: the tool that returned an error becomes **unselectable** for the rest of the run; if the draw gives it, the next tool in the list is used. - **AQ53.** There is no second tool in the set that does the same job; "moving on to another tool" means **dropping** the failed tool, and if the plan needs it, the plan cannot be completed. #### Three Policies, One Stream The block sets up the shared definition: the fictional environment, five tools, twenty-four tasks, and the loop. ```python # FICTIONAL. The measurement network's subscriber records, document set, five tools and # twenty-four tasks are defined within the lesson; no real endpoint or tool server is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "previous": previous, "current": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1) def t_read_meter(subscriber=None, **_): a = SUBSCRIBERS.get(subscriber) if a is None: return {"error": "no such subscriber"} return {"previous": a["previous"], "current": a["current"], "type": a["type"], "region": a["region"]} def t_consumption(previous=None, current=None, **_): if previous is None or current is None: return {"error": "missing field"} f = current - previous return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, type="residential", **_): if m3 is None: return {"error": "missing field"} return {"amount": _charge(m3, type)} def t_document(query=None, **_): if not query: return {"error": "missing field"} q = set(tokenize(query)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"} def t_record(subscriber=None, column=None, value=None, **_): if not (subscriber and column and value is not None): return {"error": "missing field"} return {"written": True, "subscriber": subscriber, "column": column, "value": value} TOOLS = { "read_meter": {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False}, "consumption": {"function": t_consumption, "fields": ("previous", "current"), "side_effect": False}, "tariff": {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False}, "search_docs": {"function": t_document, "fields": ("query",), "side_effect": False}, "write_record": {"function": t_record, "fields": ("subscriber", "column", "value"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"error": "no such tool"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("read_meter", "consumption", "tariff")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("read_meter", "consumption")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("search_docs",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("read_meter", "consumption", "tariff", "write_record")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["type"]) def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["approvals"] = state.get("approvals", 0) + 1 if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["result"] = s["m3"] if choice == "tariff": state["result"] = s["amount"] if choice == "search_docs": state["result"] = s["code"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} ``` The policy changes only the error branch. ```python # The recovery policy changes only the error branch of the shared loop. BUDGET, RUNS = 6, 10 RESULT_FIELD = {"consumption": "m3", "tariff": "amount", "search_docs": "code"} # tools that write the result def agent_policy(g, policy, max_steps=BUDGET, seed=SEED): r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors, banned = [], 0, 0, 0, set() seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: if r() < 0.88: break choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] while choice in banned and len(banned) < len(TOOL_NAMES): choice = TOOL_NAMES[(TOOL_NAMES.index(choice) + 1) % len(TOOL_NAMES)] if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if policy == "stop": break if policy == "skip": banned.add(choice) continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice in RESULT_FIELD: state["result"] = s[RESULT_FIELD[choice]] if choice == target: k += 1 return {"steps": step, "tokens": tok, "errors": errors, "answer": state.get("result")} FIELDS = ("steps", "tokens", "errors", "answer") print("retry is the same as the shared agent:", all(agent_policy(g, "retry") == {a: agent(g, BUDGET, SEED)[a] for a in FIELDS} for g in TASKS)) print("\nseed policy success steps errors tokens steps per correct answer") for base_seed in (20260218, 20260219): for p in ("stop", "retry", "skip"): correct, steps, tok, errors = 0, 0, 0, 0 for d in range(RUNS): for g in TASKS: s = agent_policy(g, p, BUDGET, base_seed + 7 * d) steps += s["steps"] tok += s["tokens"] errors += s["errors"] correct += s["answer"] == correct_answer(g) n = RUNS * len(TASKS) print(f"{base_seed} {p:<8} {correct / n:>8.4f} {steps:>5} {errors:>6} {tok:>9}" f" {steps / correct:>24.2f}") ``` ``` retry is the same as the shared agent: True seed policy success steps errors tokens steps per correct answer 20260218 stop 0.7500 727 53 81996 4.04 20260218 retry 0.9375 900 61 106352 4.00 20260218 skip 0.8917 907 64 108030 4.24 20260219 stop 0.7667 768 54 89180 4.17 20260219 retry 0.9542 944 66 114430 4.12 20260219 skip 0.8625 954 73 116776 4.61 ``` #### What the Policy Adds and What It Costs The first line of the output is a check: the loop we used throughout the course had an unwritten policy, and it was **retry**. **Stop** is the cheapest in all the cost columns: 727 steps, 53 tool errors, 81996 tokens. Its task success is **0.1875** below the 0.9375 of retry; since the difference between the two runs is 0.0167, the drop is ten times the band. Read per correct answer, the saving disappears: 4.04 steps for stop, 4.00 for retry, and the difference is within stop's own difference between runs (4.04 against 4.17). Stopping lowers the **spending** of steps; it does not change the **efficiency**. **Skip** is the example for the course's second claim: intuition expects the errors to decrease, and the measurement gives the opposite. The tool errors rise from 61 to **64**, and in the second run from 66 to **73**; the steps go up to 907, and the task success falls to **0.8917**. The mechanism works in two directions: the tool that gets banned has often been selected wrongly already, and the one that replaces it is wrong too, that is, the error slides onto another name; if a tool in the plan is banned, the plan ends. The loss is 0.0458 and 0.0917 in the two runs; since the widest band is 0.0292, both are above it, but the first is only one and a half times the band. #### Summary - The recovery policy is the error branch of the loop; even if it is not written, a policy runs, and here the one that runs is retry. - Stop is the cheapest (727 steps, 53 errors, 81996 tokens), lowers task success by 0.1875, and does not improve step efficiency. - Dropping the failed tool raises the tool errors to 64 and lowers task success to 0.8917. #### Course Wrap-Up | Lesson | Without scaffolding | With scaffolding | Cost | |---|---|---|---| | What Is an Agent — task success | no tools 0.0417 | loop 0.9583 (**+0.9166**) | steps from 24 to 97 | | The Agent Loop — share of right selection | depth 1: 0.8000 | depth 4: 0.5000 (**−0.3000**) | steps from 2.86 to 7.67 | | The Planner and Executor Split — task success | single loop 0.9583 | planner–executor 0.8750 (**−0.0833**) | steps from 97 to 118 | | Stopping Criteria — task success | budget 2: 0.3750 | budget 6: 1.0000 (**+0.6250**) | tokens from 4758 to 10884 | | Self-Critique and Verification — task success (24) | no rounds 21/24 | self-critique 23/24 (**+2**) | steps from 84 to 152 | | Multi-Agent Patterns — task success (24) | single agent 23/24 | two agents 23/24 (**0**) | steps from 97 to 140 | | Keeping the Human in the Loop — wrong record | approval off 3 | approval on side effects 1 (**−2**) | 7 approval points | | Tool Definition — valid call rate | name only 0.2803 | schema 0.8919 (**+0.6116**) | tool errors from 113 to 8 | | Executing Tool Calls — observation tokens | raw 770 | summarized 401 (**−369**) | task success the same: 0.9583 | | Code Execution Tools — code task success | loose limit 0.7778 | medium limit 0.3333 (**−0.4445**) | 24 stopped calls | | File System and Terminal Access — completed tasks | full authority 0.9583 | allowlist 0.8333 (**−0.1250**) | 20 stopped calls | | Web Search and Browsing — correct answer | no external knowledge 0.2083 | primary source 0.7500 (**+0.5417**) | calls from 91 to 121 | | Short-Term Memory — correct answer | unlimited window 0.9583 | window 120: 0.7083 (**−0.2500**) | tokens from 9316 to 12026 | | Long-Term Memory — steps gained (48) | no memory 0 | selective write 118 (**+118**) | 24 records | | Context Compression — total tokens | no compaction 21913 | selective 16569 (**−5344**) | 8 tasks with the field dropped | | Memory Decay — stale read share | no threshold 0.4528 | threshold 1: 0.0000 (**−0.4528**) | valid records dropped 0.9701 | | Measuring Task Success and Step Efficiency Separately — task success | toolless baseline 0.2917 | course budget 0.9583 (**+0.6666**) | per answer from 3.4286 to 4.2174 | | Testing Each Tool Separately from the Loop — defects caught (11) | untested 0 | tool level 7 (**+7**) | 0.0000 on definition defects | | Scenario-Based End-to-End Validation — defects caught (11) | tool level 7 | two layers 10 (**+3**) | residual `D11` | | Monitoring and Logging — share with cause found | result only 0.0000 | full observation 1.0000 (**+1.0000**) | tokens from 2738 to 13623 | | Cost and Latency Control — cost per step | large class 118.30 | small class 79.25 (**−39.05**) | per answer from 453.5 to 488.0 | | Error Recovery — task success (240 runs) | stop 0.7500 | retry 0.9375 (**+0.1875**) | steps from 727 to 900 | The rule of the course is written line by line in the table: **the number of an agent is the number of the whole loop, not of a single answer, and an agent whose stopping criterion is not written counts as unmeasured.** The numbers in the without-scaffolding column are not low because of a bad model; they come out low because where the loop will stop, how the tool introduces itself, what will be remembered, and what happens when a step breaks are not written. **The second claim is read from the same table too: an added step can make the work worse.** The rows that say it by name are these: Executing Tool Calls (truncation lowers 0.9583 to 0.2917 and **raises** the token to 15945), Web Search and Browsing (the secondary class lowers 0.7500 to 0.3750), Short-Term Memory (narrowing raises the token to 12026), Long-Term Memory (result only lowers 0.9167 to 0.5833), Context Compression (the tightest summarization spends 1.53 times the baseline), Multi-Agent Patterns (the second agent brings no gain), Keeping the Human in the Loop (spreading the scope raises the question to 74), Cost and Latency Control (the small class is expensive per answer), and this lesson (`skip` raises the errors to 64). In nine places the added step cost, because it was added. **Differences below the measurement band are also written by name and do not count as findings:** in What Is an Agent the baseline's single hit, in The Agent Loop the 0.0917 and 0.0160 of the neighboring depths, in The Planner and Executor Split the 0.0833 of the split, in Stopping Criteria the 0.0417 drop from six to eight, in Self-Critique and Verification the round differences, in Short-Term Memory the 0.0833 at the window of 200, in Memory Decay the gain of tightening the threshold below eight, and in this lesson the step efficiency of stop. None of them is zero, none of them is measured. In twenty-two lessons the agent was built, equipped, and measured. One thing was never asked: **every tool was defined by hand within a lesson.** The name, description, and schema of `read_meter` were written in this course and only the agent here saw them. How the same tool would be **handed** to another agent, another team, or another application, who would hold the definition, and who would break when the interface changed were not asked. The next course turns the tool definition from a dictionary inside a lesson into a **shared surface** and counts its cost.