--- title: 'Stopping Criteria' source: 'https://academia.sh/en/courses/ai-agents/stopping-criteria' course: 'AI Agents' language: en updated: '2026-10-21T05:02:26+00:00' license: 'CC BY-SA 4.0' objectives: - 'Defines the stopping criterion as a part of the loop and tells the budget limit apart from the decision that the job is done.' - 'Counts how success, steps, tool calls and tokens change as the step budget is swept over five values.' - 'Shows on the trace of a single task the mechanism of the point where enlarging the budget makes the job worse, and compares its effect with the measurement band.' - 'Does not count a fall that stays inside the measurement band as a finding and writes what is needed to measure the effect.' --- # Stopping Criteria Writing where the loop stops as the rule of the course and sweeping the step budget over the values two, four, six, eight and twelve: on 24 fictional tasks task success rises from 0.3750 to 1.0000 and then flattens at 0.9583; on the second run it is 0.4167, 0.8333, 0.9583, 0.9583, 0.9583. The fall from the sixth step to the eighth is 0.0417 and the run-to-run difference at the sixth step is also 0.0417, so the fall is inside the measurement band and cannot be declared a finding; its mechanism is shown on the trace of a single task and its effect counts as unmeasured. When the budget goes from six to twelve, success does not change while tokens rise from 10884 to 12214. Stopping by budget is not a stopping criterion but a safety limit: at the sixth step three tasks stop because the budget ran out. All three previous lessons held the same number fixed: the step budget was eight. That was a decision, its reason was not written, and all the results hang on it. In the split arrangement G05 needed three more steps to correct itself after its plan ran out; with a budget of six it would have given a wrong answer. The reverse is also true: an agent that has finished its job and cannot manage to stop can corrupt the correct result it holds. This lesson measures that decision. The step budget is swept over five values, and at each value the cost is written next to success. The lesson also sets the rule of the course: **the number of an agent is the number of the whole loop, not of a single answer, and an agent whose stopping condition is not written counts as unmeasured.** The baseline is the narrowest budget — a loop of two steps. - **AA31.** The environment, the tool set, the task set and the loop are the same definition as in the previous lessons and are **fictional**. The seed is **20260218**, the second run **20260219**. - **AA32. The stopping criterion is a part of the loop**, not a safety put on from outside. This simulator carries two criteria: **the decision that the job is done** and **the step budget**. - **AA33. The decision that the job is done is uncertain.** When no target is left in the sequence the agent stops with a **share of 0.88**; with the remaining share it calls one more tool. - **AA34. A runaway loop** is a loop that keeps taking steps although it has no target left. Every step it takes is counted. - **AA35. The step budget is not a stopping criterion but a hard limit.** When the budget runs out the loop is cut off midway, and whatever value it holds is the answer. - **AA36. The swept values are 2, 4, 6, 8 and 12**; everything else is fixed. - **AA37.** The counted units are **success, steps, tool calls, tokens, tool errors, runs that end by their own decision, and wandering steps**. - **AA38. The task set is 24 tasks**, so the smallest step of success is **0.0417**. A difference smaller than this cannot be represented in this set. - **AA39.** A difference below the run-to-run difference **counts as unmeasured**; this lesson applies the rule to its own result as well. - **AA40.** Time is not written. ## What a Stopping Criterion Is When a loop will stop is as much a design decision as what it will do. There are two things to tell apart. **The decision that the job is done** is the agent's own decision: no target is left in line, there is a result in hand, stop. **The step budget** is a hard limit put on from outside and is not a decision; it cuts the loop wherever it happens to be. The second cannot take the place of the first, because a loop that is cut off hands over the value it holds as the answer, and the correctness of that value has not been looked at. The classes of criteria are not limited to these: a repeated step, a token budget and an external cancellation are also stopping criteria. This simulator carries only the first two, and only their numbers are printed in this lesson. ```python # SIMULATOR -- FICTIONAL. No real model endpoint, tool server or runtime # is called; the environment, the tools and the tasks are all fictional. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL environment: records of the measurement network REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "previous": previous, "current": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1) def t_read_meter(subscriber=None, **_): a = SUBSCRIBERS.get(subscriber) if a is None: return {"error": "no such subscriber"} return {"previous": a["previous"], "current": a["current"], "type": a["type"], "region": a["region"]} def t_consumption(previous=None, current=None, **_): if previous is None or current is None: return {"error": "missing field"} f = current - previous return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, type="residential", **_): if m3 is None: return {"error": "missing field"} return {"amount": _charge(m3, type)} def t_document(query=None, **_): if not query: return {"error": "missing field"} q = set(tokenize(query)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"} def t_record(subscriber=None, column=None, value=None, **_): if not (subscriber and column and value is not None): return {"error": "missing field"} return {"written": True, "subscriber": subscriber, "column": column, "value": value} TOOLS = { "read_meter": {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False}, "consumption": {"function": t_consumption, "fields": ("previous", "current"), "side_effect": False}, "tariff": {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False}, "search_docs": {"function": t_document, "fields": ("query",), "side_effect": False}, "write_record": {"function": t_record, "fields": ("subscriber", "column", "value"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) def call_tool(name, state): """Fields outside the schema are filtered out; a missing field is the tool's own error.""" CALLS[0] += 1 if name not in TOOLS: return {"error": "no such tool"} kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state} return TOOLS[name]["function"](**kw) # ---- FICTIONAL tasks: each one is solved by a known tool sequence TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("read_meter", "consumption", "tariff")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("read_meter", "consumption")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("search_docs",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("read_meter", "consumption", "tariff", "write_record")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["current"] - a["previous"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["type"]) # ---- FICTIONAL agent: at each step it picks a tool or finishes def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False): """FICTIONAL agent loop. No real model is called; tool selection comes from a single draw that depends on the task's sequence and the depth.""" r = rng(seed ^ trace_id(g["name"])) state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"} trace, step, tok, errors = [], 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) target = seq[k] if k < len(seq) else None if target is None: # the job is done , the agent should stop if r() < 0.88: return {"done": True, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if TOOLS[choice]["side_effect"] and approval: state["approvals"] = state.get("approvals", 0) + 1 if choice == "write_record": state["value"] = state.get("result") s = call_tool(choice, state) trace.append((choice, "error" in s)) tok += 30 if "error" in s: errors += 1 if self_critique: tok += 40 continue state.update({kk: vv for kk, vv in s.items() if kk != "error"}) if choice == "consumption": state["result"] = s["m3"] if choice == "tariff": state["result"] = s["amount"] if choice == "search_docs": state["result"] = s["code"] if choice == target: k += 1 return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors, "answer": state.get("result"), "approvals": state.get("approvals", 0)} def wandering(g, s): """Number of steps taken after the job is done: tool calls made although no target is left. Every step for which no decision to stop was made is written here.""" k = n = 0 for choice, failed in s["trace"]: if k >= len(g["sequence"]): n += 1 elif choice == g["sequence"][k] and not failed: k += 1 return n def sweep(b, seed=SEED): CALLS[0] = 0 success = step = tok = error = done = wander = 0 for x in TASKS: y = agent(x, max_steps=b, seed=seed) step, tok, error = step + y["steps"], tok + y["tokens"], error + y["errors"] success += y["answer"] == correct_answer(x) done += y["done"] wander += wandering(x, y) return success / len(TASKS), step, CALLS[0], tok, error, done, wander for label, seed in (("A", SEED), ("B", SEED + 1)): print(f"run {label} budget success step calls tokens error done wandering") for b in (2, 4, 6, 8, 12): z = sweep(b, seed) print(f" {b:>6} {z[0]:>8.4f} {z[1]:>6} {z[2]:>6} {z[3]:>9}" f" {z[4]:>5} {z[5]:>6} {z[6]:>9}") ``` ``` run A budget success step calls tokens error done wandering 2 0.3750 48 45 4758 8 3 1 4 0.8750 84 64 9402 8 20 1 6 1.0000 92 71 10884 8 21 1 8 0.9583 97 74 11978 8 23 1 12 0.9583 98 74 12214 8 24 1 run B budget success step calls tokens error done wandering 2 0.4167 48 45 4758 7 3 2 4 0.8333 83 67 9366 7 16 3 6 0.9583 96 74 11610 7 22 3 8 0.9583 98 74 11994 7 24 3 12 0.9583 98 74 11994 7 24 3 ``` The first three rows are beyond dispute. The budget of two steps gives sequences of three and four steps no chance at all; success can come only from the short tasks and stays at **0.3750** and **0.4167**. At the fourth step it is 0.8750 and 0.8333, at the sixth step **1.0000** and **0.9583**. The gain from two to six is 0.6250 and 0.5416; since the widest run-to-run difference is **0.0417**, this gain is more than ten times the band and is **measured**. The gain from four to six is 0.1250 on both runs, and that too is above the band. The `done` column changes the nature of this gain. With the budget of two steps only **three** of the twenty-four runs stop by their own decision; the other twenty-one are cut off because the budget ran out. Even at the sixth step three runs end by being cut off. A run that is cut off has not produced an answer; it has handed over the value it holds. **The budget is not a stopping criterion but a limit of last resort in the absence of one.** ## From the Sixth Step to the Eighth: The Mechanism Exists, the Measure Does Not The striking place in the table is the move from the sixth row to the eighth. On the first run success falls **from 1.0000 to 0.9583**. A single task is responsible for the fall, and its trace can be read. ```python g = [x for x in TASKS if x["name"] == "G23"][0] print(f"{g['name']} ({g['kind']}) sequence :", " , ".join(g["sequence"]), "| correct answer", correct_answer(g)) for b in (6, 8, 12): s = agent(g, max_steps=b) print(f" budget {b:>2} step {s['steps']} done {s['done']} answer {s['answer']}") print(" trace :", " , ".join(a for a, _ in s["trace"])) ``` ``` G23 (record) sequence : read_meter , consumption , tariff , write_record | correct answer 1132 budget 6 step 6 done False answer 1132 trace : read_meter , consumption , search_docs , consumption , tariff , tariff budget 8 step 8 done False answer TR01 trace : read_meter , consumption , search_docs , consumption , tariff , tariff , search_docs , write_record budget 12 step 9 done True answer TR01 trace : read_meter , consumption , search_docs , consumption , tariff , tariff , search_docs , write_record ``` With the budget of six steps G23 gives the right answer, but **not by its own decision**: the budget runs out at the sixth step and the value in the state at that moment is 1132, which is correct. The budget of eight steps allows two more steps. At the seventh step the agent picks `search_docs`; this choice is wrong but **gives no error**, because the field the tool asks for is in the state. The call, returning successfully, writes a document code over the accumulated result. At the eighth step the target is finally hit and `write_record` puts that corrupted value into the record. The budget of twelve steps also makes the decision to stop at the ninth step, but there is nothing left to correct. The mechanism is clear and can be written in one sentence: **once the correct result has been written to the state, every extra step is a chance to corrupt it.** The second channel of the loop looks the same way — when the job is done the agent does not stop with a share of 0.12 and calls one more tool; the number of wandering steps in these runs is 1 and 3. On the other hand, **this fall is not a finding**. The numbers the two runs give at the sixth step are 1.0000 and 0.9583, so the run-to-run difference at that point is **0.0417**. The fall from six to eight is also exactly 0.0417, and on the second run it is **zero**. The difference stays inside the measurement band: **the mechanism can be shown, its effect counts as unmeasured.** This is a limit imposed by scale. In a set of twenty-four tasks a task is worth 0.0417, and the band is also one task wide, so an effect of one task can never rise above the band in this set. The reader who wants to measure the effect has two ways. The first is **enlarging the task set**: if the set grows to several times its size, the resolution increases. The second is **sweeping the wandering share as a setting**; if 0.12 stops being a single point and turns into an axis, how success changes along that axis can be read without depending on the difference of a single task. What is measured is the cost. When the budget goes from six to twelve, success does not change on the first run, whereas tokens rise **from 10884 to 12214**, steps from 92 to 98 and tool calls from 71 to 74. The move from eight to twelve leaves success fixed on both runs. **The gain of enlarging the budget could not be measured; its cost was measured.** ## Summary - The stopping criterion is a part of the loop; the decision that the job is done is the agent's own decision, the step budget is a hard limit put on from outside, and it cannot take the place of the first. - Success is 0.3750 and 0.4167 at two steps and 1.0000 and 0.9583 at six steps; the gain from two to six is more than ten times the widest run-to-run difference and is measured. - With the budget of two steps only three of the twenty-four runs stop by their own decision; even at six steps three runs are cut off because the budget ran out. - The fall from six to eight is 0.0417 and the run-to-run difference at the sixth step is also 0.0417; the mechanism can be shown on the trace of a single task, but its effect counts as unmeasured. - Measuring the effect requires enlarging the task set or sweeping the wandering share on its own axis. - When the budget goes from six to twelve, success does not change while tokens rise from 10884 to 12214: the gain could not be measured, the cost was measured. ## Next Step In four lessons the loop was built, opened up step by step, split in two, and where it stops was written. All these measurements had a common support, and that support was not questioned: **we** knew the correct answer. The task set carried the correct answer of every task, and we calculated success by comparing against that list. When the agent itself stopped, it never looked once at the value in its hand; G23 put a corrupted result into the record and did not see it as a problem. The next lesson enters this gap: if the agent adds a round that checks its own output, what does it catch, and how many steps and how many tokens does that round cost.