Stopping Criteria

All three previous lessons held the same number fixed: the step budget was eight. That was a decision, its reason was not written, and all the results hang on it. In the split arrangement G05 needed three more steps to correct itself after its plan ran out; with a budget of six it would have given a wrong answer. The reverse is also true: an agent that has finished its job and cannot manage to stop can corrupt the correct result it holds.

This lesson measures that decision. The step budget is swept over five values, and at each value the cost is written next to success. The lesson also sets the rule of the course: the number of an agent is the number of the whole loop, not of a single answer, and an agent whose stopping condition is not written counts as unmeasured. The baseline is the narrowest budget — a loop of two steps.

• AA31. The environment, the tool set, the task set and the loop are the same definition as in the previous lessons and are fictional. The seed is 20260218, the second run 20260219.
• AA32. The stopping criterion is a part of the loop, not a safety put on from outside. This simulator carries two criteria: the decision that the job is done and the step budget.
• AA33. The decision that the job is done is uncertain. When no target is left in the sequence the agent stops with a share of 0.88; with the remaining share it calls one more tool.
• AA34. A runaway loop is a loop that keeps taking steps although it has no target left. Every step it takes is counted.
• AA35. The step budget is not a stopping criterion but a hard limit. When the budget runs out the loop is cut off midway, and whatever value it holds is the answer.
• AA36. The swept values are 2, 4, 6, 8 and 12; everything else is fixed.
• AA37. The counted units are success, steps, tool calls, tokens, tool errors, runs that end by their own decision, and wandering steps.
• AA38. The task set is 24 tasks, so the smallest step of success is 0.0417. A difference smaller than this cannot be represented in this set.
• AA39. A difference below the run-to-run difference counts as unmeasured; this lesson applies the rule to its own result as well.
• AA40. Time is not written.

What a Stopping Criterion Is

When a loop will stop is as much a design decision as what it will do. There are two things to tell apart. The decision that the job is done is the agent’s own decision: no target is left in line, there is a result in hand, stop. The step budget is a hard limit put on from outside and is not a decision; it cuts the loop wherever it happens to be. The second cannot take the place of the first, because a loop that is cut off hands over the value it holds as the answer, and the correctness of that value has not been looked at.

The classes of criteria are not limited to these: a repeated step, a token budget and an external cancellation are also stopping criteria. This simulator carries only the first two, and only their numbers are printed in this lesson.

# SIMULATOR -- FICTIONAL. No real model endpoint, tool server or runtime
# is called; the environment, the tools and the tasks are all fictional.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in str(m).lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def trace_id(m):
    h = 2166136261
    for c in str(m):
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


# ---- FICTIONAL environment: records of the measurement network
REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
SUBSCRIBERS = {}
for i in range(60):
    r = rng(SEED + 31 * i)
    previous = 10000 + int(r() * 80000)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "previous": previous, "current": previous + int(r() * 70)}
TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16
COMMERCIAL_FACTOR = 2
DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu",
             "TR02": "isyeri abonesinde birim ucret iki katidir",
             "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir",
             "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider",
             "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"}

CALLS = [0]                              # tool call counter


def _charge(m3, type_):
    remaining, lower, total = m3, 0, 0
    for upper, unit in TIERS:
        band = min(remaining, upper - lower)
        if band <= 0:
            break
        total += band * unit
        remaining -= band
        lower = upper
    total += remaining * ABOVE
    return total * (COMMERCIAL_FACTOR if type_ == "commercial" else 1)


def t_read_meter(subscriber=None, **_):
    a = SUBSCRIBERS.get(subscriber)
    if a is None:
        return {"error": "no such subscriber"}
    return {"previous": a["previous"], "current": a["current"], "type": a["type"],
            "region": a["region"]}


def t_consumption(previous=None, current=None, **_):
    if previous is None or current is None:
        return {"error": "missing field"}
    f = current - previous
    return {"m3": f + 100000 if f < 0 else f}


def t_tariff(m3=None, type="residential", **_):
    if m3 is None:
        return {"error": "missing field"}
    return {"amount": _charge(m3, type)}


def t_document(query=None, **_):
    if not query:
        return {"error": "missing field"}
    q = set(tokenize(query))
    best, code = -1, None
    for k, m in DOCUMENTS.items():
        o = len(q & set(tokenize(m)))
        if o > best:
            best, code = o, k
    return {"code": code, "text": DOCUMENTS[code]} if best > 0 else {"error": "not found"}


def t_record(subscriber=None, column=None, value=None, **_):
    if not (subscriber and column and value is not None):
        return {"error": "missing field"}
    return {"written": True, "subscriber": subscriber, "column": column, "value": value}


TOOLS = {
    "read_meter":    {"function": t_read_meter, "fields": ("subscriber",), "side_effect": False},
    "consumption":   {"function": t_consumption, "fields": ("previous", "current"),
                      "side_effect": False},
    "tariff":        {"function": t_tariff, "fields": ("m3", "type"), "side_effect": False},
    "search_docs":   {"function": t_document, "fields": ("query",), "side_effect": False},
    "write_record":  {"function": t_record, "fields": ("subscriber", "column", "value"),
                      "side_effect": True},
}
TOOL_NAMES = list(TOOLS)


def call_tool(name, state):
    """Fields outside the schema are filtered out; a missing field is the tool's own error."""
    CALLS[0] += 1
    if name not in TOOLS:
        return {"error": "no such tool"}
    kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state}
    return TOOLS[name]["function"](**kw)


# ---- FICTIONAL tasks: each one is solved by a known tool sequence
TASKS = []
for i in range(24):
    r = rng(SEED + 9973 * i)
    sub = f"A{1 + int(r() * 60):03d}"
    t = pick(r(), [0.30, 0.22, 0.20, 0.28])
    if t == 0:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub,
                      "request": f"{sub} abonesinin bu donem bedeli nedir",
                      "sequence": ("read_meter", "consumption", "tariff")})
    elif t == 1:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub,
                      "request": f"{sub} abonesinin bu donem tuketimi kac metrekup",
                      "sequence": ("read_meter", "consumption")})
    elif t == 2:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub,
                      "request": "sayac donusunde endeks farki nasil hesaplanir",
                      "sequence": ("search_docs",)})
    else:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub,
                      "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz",
                      "sequence": ("read_meter", "consumption", "tariff", "write_record")})


def correct_answer(g):
    a = SUBSCRIBERS[g["subscriber"]]
    f = a["current"] - a["previous"]
    m3 = f + 100000 if f < 0 else f
    if g["kind"] == "consumption":
        return m3
    if g["kind"] == "document":
        return "SY01"
    return _charge(m3, a["type"])


# ---- FICTIONAL agent: at each step it picks a tool or finishes
def selection(d):
    return max(0.34, 0.80 - 0.09 * (d - 1))       # share of picking the right tool, falls with depth


def agent(g, max_steps=8, seed=SEED, self_critique=False, approval=False):
    """FICTIONAL agent loop. No real model is called; tool selection
    comes from a single draw that depends on the task's sequence and the depth."""
    r = rng(seed ^ trace_id(g["name"]))
    state = {"subscriber": g["subscriber"], "query": g["request"], "column": "charge"}
    trace, step, tok, errors = [], 0, 0, 0
    seq, k = g["sequence"], 0
    while step < max_steps:
        step += 1
        tok += 60 + 22 * len(trace)
        target = seq[k] if k < len(seq) else None
        if target is None:                         # the job is done , the agent should stop
            if r() < 0.88:
                return {"done": True, "steps": step, "trace": trace, "tokens": tok,
                        "errors": errors, "answer": state.get("result"),
                        "approvals": state.get("approvals", 0)}
            choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))]
        else:
            choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))]
        if TOOLS[choice]["side_effect"] and approval:
            state["approvals"] = state.get("approvals", 0) + 1
        if choice == "write_record":
            state["value"] = state.get("result")
        s = call_tool(choice, state)
        trace.append((choice, "error" in s))
        tok += 30
        if "error" in s:
            errors += 1
            if self_critique:
                tok += 40
            continue
        state.update({kk: vv for kk, vv in s.items() if kk != "error"})
        if choice == "consumption":
            state["result"] = s["m3"]
        if choice == "tariff":
            state["result"] = s["amount"]
        if choice == "search_docs":
            state["result"] = s["code"]
        if choice == target:
            k += 1
    return {"done": False, "steps": step, "trace": trace, "tokens": tok, "errors": errors,
            "answer": state.get("result"), "approvals": state.get("approvals", 0)}



def wandering(g, s):
    """Number of steps taken after the job is done: tool calls made although
    no target is left. Every step for which no decision to stop was made is written here."""
    k = n = 0
    for choice, failed in s["trace"]:
        if k >= len(g["sequence"]):
            n += 1
        elif choice == g["sequence"][k] and not failed:
            k += 1
    return n


def sweep(b, seed=SEED):
    CALLS[0] = 0
    success = step = tok = error = done = wander = 0
    for x in TASKS:
        y = agent(x, max_steps=b, seed=seed)
        step, tok, error = step + y["steps"], tok + y["tokens"], error + y["errors"]
        success += y["answer"] == correct_answer(x)
        done += y["done"]
        wander += wandering(x, y)
    return success / len(TASKS), step, CALLS[0], tok, error, done, wander


for label, seed in (("A", SEED), ("B", SEED + 1)):
    print(f"run {label}     budget  success   step  calls    tokens error   done wandering")
    for b in (2, 4, 6, 8, 12):
        z = sweep(b, seed)
        print(f"          {b:>6} {z[0]:>8.4f} {z[1]:>6} {z[2]:>6} {z[3]:>9}"
              f" {z[4]:>5} {z[5]:>6} {z[6]:>9}")

run A     budget  success   step  calls    tokens error   done wandering
               2   0.3750     48     45      4758     8      3         1
               4   0.8750     84     64      9402     8     20         1
               6   1.0000     92     71     10884     8     21         1
               8   0.9583     97     74     11978     8     23         1
              12   0.9583     98     74     12214     8     24         1
run B     budget  success   step  calls    tokens error   done wandering
               2   0.4167     48     45      4758     7      3         2
               4   0.8333     83     67      9366     7     16         3
               6   0.9583     96     74     11610     7     22         3
               8   0.9583     98     74     11994     7     24         3
              12   0.9583     98     74     11994     7     24         3

The first three rows are beyond dispute. The budget of two steps gives sequences of three and four steps no chance at all; success can come only from the short tasks and stays at 0.3750 and 0.4167. At the fourth step it is 0.8750 and 0.8333, at the sixth step 1.0000 and 0.9583. The gain from two to six is 0.6250 and 0.5416; since the widest run-to-run difference is 0.0417, this gain is more than ten times the band and is measured. The gain from four to six is 0.1250 on both runs, and that too is above the band.

The done column changes the nature of this gain. With the budget of two steps only three of the twenty-four runs stop by their own decision; the other twenty-one are cut off because the budget ran out. Even at the sixth step three runs end by being cut off. A run that is cut off has not produced an answer; it has handed over the value it holds. The budget is not a stopping criterion but a limit of last resort in the absence of one.

From the Sixth Step to the Eighth: The Mechanism Exists, the Measure Does Not

The striking place in the table is the move from the sixth row to the eighth. On the first run success falls from 1.0000 to 0.9583. A single task is responsible for the fall, and its trace can be read.

g = [x for x in TASKS if x["name"] == "G23"][0]
print(f"{g['name']} ({g['kind']}) sequence :", " , ".join(g["sequence"]),
      "| correct answer", correct_answer(g))
for b in (6, 8, 12):
    s = agent(g, max_steps=b)
    print(f"  budget {b:>2}  step {s['steps']}  done {s['done']}  answer {s['answer']}")
    print("     trace :", " , ".join(a for a, _ in s["trace"]))

G23 (record) sequence : read_meter , consumption , tariff , write_record | correct answer 1132
  budget  6  step 6  done False  answer 1132
     trace : read_meter , consumption , search_docs , consumption , tariff , tariff
  budget  8  step 8  done False  answer TR01
     trace : read_meter , consumption , search_docs , consumption , tariff , tariff , search_docs , write_record
  budget 12  step 9  done True  answer TR01
     trace : read_meter , consumption , search_docs , consumption , tariff , tariff , search_docs , write_record

With the budget of six steps G23 gives the right answer, but not by its own decision: the budget runs out at the sixth step and the value in the state at that moment is 1132, which is correct. The budget of eight steps allows two more steps. At the seventh step the agent picks search_docs; this choice is wrong but gives no error, because the field the tool asks for is in the state. The call, returning successfully, writes a document code over the accumulated result. At the eighth step the target is finally hit and write_record puts that corrupted value into the record. The budget of twelve steps also makes the decision to stop at the ninth step, but there is nothing left to correct.

The mechanism is clear and can be written in one sentence: once the correct result has been written to the state, every extra step is a chance to corrupt it. The second channel of the loop looks the same way — when the job is done the agent does not stop with a share of 0.12 and calls one more tool; the number of wandering steps in these runs is 1 and 3.

On the other hand, this fall is not a finding. The numbers the two runs give at the sixth step are 1.0000 and 0.9583, so the run-to-run difference at that point is 0.0417. The fall from six to eight is also exactly 0.0417, and on the second run it is zero. The difference stays inside the measurement band: the mechanism can be shown, its effect counts as unmeasured.

This is a limit imposed by scale. In a set of twenty-four tasks a task is worth 0.0417, and the band is also one task wide, so an effect of one task can never rise above the band in this set. The reader who wants to measure the effect has two ways. The first is enlarging the task set: if the set grows to several times its size, the resolution increases. The second is sweeping the wandering share as a setting; if 0.12 stops being a single point and turns into an axis, how success changes along that axis can be read without depending on the difference of a single task.

What is measured is the cost. When the budget goes from six to twelve, success does not change on the first run, whereas tokens rise from 10884 to 12214, steps from 92 to 98 and tool calls from 71 to 74. The move from eight to twelve leaves success fixed on both runs. The gain of enlarging the budget could not be measured; its cost was measured.

Summary

• The stopping criterion is a part of the loop; the decision that the job is done is the agent’s own decision, the step budget is a hard limit put on from outside, and it cannot take the place of the first.
• Success is 0.3750 and 0.4167 at two steps and 1.0000 and 0.9583 at six steps; the gain from two to six is more than ten times the widest run-to-run difference and is measured.
• With the budget of two steps only three of the twenty-four runs stop by their own decision; even at six steps three runs are cut off because the budget ran out.
• The fall from six to eight is 0.0417 and the run-to-run difference at the sixth step is also 0.0417; the mechanism can be shown on the trace of a single task, but its effect counts as unmeasured.
• Measuring the effect requires enlarging the task set or sweeping the wandering share on its own axis.
• When the budget goes from six to twelve, success does not change while tokens rise from 10884 to 12214: the gain could not be measured, the cost was measured.

Next Step

In four lessons the loop was built, opened up step by step, split in two, and where it stops was written. All these measurements had a common support, and that support was not questioned: we knew the correct answer. The task set carried the correct answer of every task, and we calculated success by comparing against that list. When the agent itself stopped, it never looked once at the value in its hand; G23 put a corrupted result into the record and did not see it as a problem. The next lesson enters this gap: if the agent adds a round that checks its own output, what does it catch, and how many steps and how many tokens does that round cost.