Executing Tool Calls

In the previous lesson the tool definition was split into four levels, and at the highest level the valid call rate was 0.8919 and task success 0.9583. That measurement never questioned the return of the call: whatever the tool returned was written to the state with all its fields and carried to the next step.

In a loop this is a shaky assumption. A tool result is an observation given back to the model, and that observation sits in the prompt again at every step. The full text a document tool returns, the four fields a reading tool returns, the confirmation of a write tool: all of them accumulate as the loop grows. The question of this lesson is this: in what form should the observation be given back, what happens to task success, and how many tokens accumulate across the loop.

• TM9. The environment, the tool set, the task set, and the agent loop are fictional; no real model endpoint, tool server, or runtime is called. 60 fictional subscribers, 5 fictional documents, 5 tools, and 24 tasks each solved by a known tool sequence. Seed 20260218, second run 20260219.
• TM10. A tool returns a dictionary; the observation is that dictionary turned into text. The agent updates its state not from the dictionary but by reading back from the observation text.
• TM11. There are three observation forms. The raw observation carries all the fields. The truncated observation cuts the text at a given number of words; a field whose value is left incomplete at the cut cannot be read back. The summarized observation carries the fields declared per tool.
• TM12. Truncation is measured at two thresholds: seven words and four words.
• TM13. The accumulated observation is sent again in the prompt at every step. The shared loop counts this accumulation with a constant coefficient per step; this lesson puts the token count of the observation actually carried in place of that coefficient. Tokens are counted with the fictional tokenizer.
• TM14. The observation of a call that returns an error also accumulates; an error text is also an observation.
• TM15. The baseline is the raw observation; this is the loop without scaffolding. Truncation and summarizing are the scaffolding added. The step budget is eight in all four forms.
• TM16. Task success is the result the loop leaves being equal to the correct answer.

The Three Forms of Observation

The code below builds the shared environment, the tool set, and the task set, then adds the observation layer: a writer that turns the dictionary into text, a parser that reads the fields back from the text, and the three forms. The tool and field names in the code are the read_meter, consumption, tariff, search_docs, and write_record tools of the previous lessons; they are counted in tokens here, so they stay Turkish.

SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in str(m).lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def trace_id(m):
    h = 2166136261
    for c in str(m):
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


# ---- FICTIONAL environment: records of the measurement network
# wire words stay Turkish: they are counted in tokens (same rule as the MCP course)
# onceki/simdiki/tip/bolge = previous/current/type/region; mesken/isyeri = residential/commercial
REGION = ["dogu", "bati", "kuzey", "guney", "merkez"]
TYPE = ["mesken", "isyeri"]
SUBSCRIBERS = {}
for i in range(60):
    r = rng(SEED + 31 * i)
    previous = 10000 + int(r() * 80000)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"bolge": REGION[int(r() * 5)], "tip": TYPE[int(r() * 2)],
                                    "onceki": previous, "simdiki": previous + int(r() * 70)}
TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16
COMMERCIAL_FACTOR = 2
DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu",
             "TR02": "isyeri abonesinde birim ucret iki katidir",
             "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir",
             "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider",
             "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"}

CALLS = [0]                              # tool call counter


def _charge(m3, type_):
    remaining, lower, total = m3, 0, 0
    for upper, unit in TIERS:
        band = min(remaining, upper - lower)
        if band <= 0:
            break
        total += band * unit
        remaining -= band
        lower = upper
    total += remaining * ABOVE
    return total * (COMMERCIAL_FACTOR if type_ == "isyeri" else 1)


def t_read_meter(abone=None, **_):
    a = SUBSCRIBERS.get(abone)
    if a is None:
        return {"hata": "abone yok"}
    return {"onceki": a["onceki"], "simdiki": a["simdiki"], "tip": a["tip"],
            "bolge": a["bolge"]}


def t_consumption(onceki=None, simdiki=None, **_):
    if onceki is None or simdiki is None:
        return {"hata": "eksik alan"}
    f = simdiki - onceki
    return {"m3": f + 100000 if f < 0 else f}


def t_tariff(m3=None, tip="mesken", **_):
    if m3 is None:
        return {"hata": "eksik alan"}
    return {"tutar": _charge(m3, tip)}


def t_document(anahtar=None, **_):
    if not anahtar:
        return {"hata": "eksik alan"}
    q = set(tokenize(anahtar))
    best, code = -1, None
    for k, m in DOCUMENTS.items():
        o = len(q & set(tokenize(m)))
        if o > best:
            best, code = o, k
    return {"kod": code, "metin": DOCUMENTS[code]} if best > 0 else {"hata": "bulunamadi"}


def t_record(abone=None, sutun=None, deger=None, **_):
    if not (abone and sutun and deger is not None):
        return {"hata": "eksik alan"}
    return {"yazildi": True, "abone": abone, "sutun": sutun, "deger": deger}


TOOLS = {
    "sayac_oku":  {"function": t_read_meter, "fields": ("abone",), "side_effect": False},
    "tuketim":    {"function": t_consumption, "fields": ("onceki", "simdiki"),
                      "side_effect": False},
    "tarife":     {"function": t_tariff, "fields": ("m3", "tip"), "side_effect": False},
    "belge_ara":  {"function": t_document, "fields": ("anahtar",), "side_effect": False},
    "kayit_yaz":  {"function": t_record, "fields": ("abone", "sutun", "deger"),
                      "side_effect": True},
}
TOOL_NAMES = list(TOOLS)


def call_tool(name, state):
    """Fields outside the schema are filtered out; a missing field is the tool's own error."""
    CALLS[0] += 1
    if name not in TOOLS:
        return {"hata": "arac yok"}
    kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state}
    return TOOLS[name]["function"](**kw)


# ---- FICTIONAL tasks: each one is solved by a known tool sequence
TASKS = []
for i in range(24):
    r = rng(SEED + 9973 * i)
    sub = f"A{1 + int(r() * 60):03d}"
    t = pick(r(), [0.30, 0.22, 0.20, 0.28])
    if t == 0:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub,
                      "request": f"{sub} abonesinin bu donem bedeli nedir",
                      "sequence": ("sayac_oku", "tuketim", "tarife")})
    elif t == 1:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub,
                      "request": f"{sub} abonesinin bu donem tuketimi kac metrekup",
                      "sequence": ("sayac_oku", "tuketim")})
    elif t == 2:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub,
                      "request": "sayac donusunde endeks farki nasil hesaplanir",
                      "sequence": ("belge_ara",)})
    else:
        TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub,
                      "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz",
                      "sequence": ("sayac_oku", "tuketim", "tarife", "kayit_yaz")})


def correct_answer(g):
    a = SUBSCRIBERS[g["subscriber"]]
    f = a["simdiki"] - a["onceki"]
    m3 = f + 100000 if f < 0 else f
    if g["kind"] == "consumption":
        return m3
    if g["kind"] == "document":
        return "SY01"
    return _charge(m3, a["tip"])


# ---- FICTIONAL agent: at each step it picks a tool or finishes
def selection(d):
    return max(0.34, 0.80 - 0.09 * (d - 1))       # share of picking the right tool, falls with depth


# The dictionary a tool returns is turned into text before it is given back to the model.
SUMMARY_FIELDS = {"sayac_oku": ("onceki", "simdiki", "tip"), "tuketim": ("m3",),
                  "tarife": ("tutar",), "belge_ara": ("kod",), "kayit_yaz": ("yazildi",)}
FORMAT = {"raw": None, "truncated 7": 7, "truncated 4": 4, "summarized": None}


def to_text(s):
    return " , ".join(f"{k} = {v}" for k, v in s.items())


def parse(text):
    """Reads the fields back from the observation text; a field whose value was cut off is dropped."""
    d = {}
    for p in text.split(" , "):
        a, _, v = p.partition(" = ")
        v = v.strip(" ,")
        if v:
            d[a.strip(" ,")] = int(v) if v.lstrip("-").isdigit() else v
    return d


def observe(name, result, form):
    """Raw , truncated , or summarized observation text."""
    if form == "summarized":
        result = {k: v for k, v in result.items() if k in SUMMARY_FIELDS.get(name, ())}
    text = to_text(result)
    if FORMAT[form]:
        text = " ".join(text.split()[:FORMAT[form]])
    return text


def agent_observation(g, form="raw", max_steps=8, seed=SEED):
    """The shared agent loop; the only differences are that the observation is turned
    into text and read back, and that the accumulated observation tokens are sent
    again at every step."""
    r = rng(seed ^ trace_id(g["name"]))
    state = {"abone": g["subscriber"], "anahtar": g["request"], "sutun": "bedel"}
    trace, step, tok, errors, accum = [], 0, 0, 0, 0
    seq, k = g["sequence"], 0
    while step < max_steps:
        step += 1
        tok += 60 + accum                    # the accumulated observation is sent again at every step
        target = seq[k] if k < len(seq) else None
        if target is None:
            if r() < 0.88:
                break
            choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))]
        else:
            choice = target if r() < selection(k + 1) else TOOL_NAMES[int(r() * len(TOOL_NAMES))]
        if choice == "kayit_yaz":
            state["deger"] = state.get("sonuc")
        s = call_tool(choice, state)
        trace.append((choice, "hata" in s))
        tok += 30
        text = observe(choice, s, form)
        accum += len(tokenize(text))
        if "hata" in s:
            errors += 1
            continue
        d = parse(text)
        state.update(d)
        if choice == "tuketim" and "m3" in d:
            state["sonuc"] = d["m3"]
        if choice == "tarife" and "tutar" in d:
            state["sonuc"] = d["tutar"]
        if choice == "belge_ara" and "kod" in d:
            state["sonuc"] = d["kod"]
        if choice == target:
            k += 1
    return {"steps": step, "tokens": tok, "errors": errors, "observation": accum,
            "answer": state.get("sonuc")}


for name in ("sayac_oku", "belge_ara"):
    sample = TOOLS[name]["function"](**{"abone": "A007", "anahtar": "endeks farki"})
    for form in ("raw", "truncated 7", "summarized"):
        m = observe(name, sample, form)
        print(f"{name:<10} {form:<11} {len(tokenize(m)):>2} tokens  {m}")
    print(f"{name:<10} truncated 7 fields read back :",
          " , ".join(parse(observe(name, sample, "truncated 7"))))

sayac_oku  raw         19 tokens  onceki = 78189 , simdiki = 78241 , tip = mesken , bolge = merkez
sayac_oku  truncated 7  9 tokens  onceki = 78189 , simdiki = 78241
sayac_oku  summarized  14 tokens  onceki = 78189 , simdiki = 78241 , tip = mesken
sayac_oku  truncated 7 fields read back : onceki , simdiki
belge_ara  raw         19 tokens  kod = SY01 , metin = endeks farki tuketimi verir sayac donusu 100000 eklenir
belge_ara  truncated 7  8 tokens  kod = SY01 , metin = endeks
belge_ara  summarized   3 tokens  kod = SY01
belge_ara  truncated 7 fields read back : kod , metin

Both tools carry the same raw length, nineteen tokens, and the two forms cut them in very different places. Truncation looks at position: in the sayac_oku (read meter) observation the last two fields, tip (type) and bolge (region), drop because they remain after the seventh word. The summary looks at declaration: in the same tool the type stays because it is declared, and the region goes because it is not. In the document tool the order is reversed; there truncation keeps the code because it comes first, while the summary throws away the whole text and drops the nineteen tokens to three.

The Numbers of the Four Forms

The same 24 fictional tasks run in four forms. Observation tokens are the total length of the observation texts produced across the loop; total tokens also include their being sent again.

def measure(form, seed=SEED):
    CALLS[0] = 0
    ok = steps = tok = errors = obs = 0
    for g in TASKS:
        s = agent_observation(g, form, seed=seed)
        ok += s["answer"] == correct_answer(g)
        steps += s["steps"]
        tok += s["tokens"]
        errors += s["errors"]
        obs += s["observation"]
    return ok / len(TASKS), CALLS[0], errors, steps, obs, tok


print("observation form  calls errors steps observation tokens total tokens"
      "  task success A  task success B  tokens per answer")
for form in FORMAT:
    a, c = measure(form), measure(form, SEED + 1)
    print(f"{form:<16} {a[1]:>6} {a[2]:>6} {a[3]:>5} {a[4]:>18} {a[5]:>12}"
          f"  {a[0]:>14.4f}  {c[0]:>14.4f}  {a[5] / (a[0] * 24):>17.0f}")
SUMMARY_FIELDS["sayac_oku"] = ("onceki", "simdiki")     # type dropped from the declaration
a = measure("summarized")
print(f"narrow summary ( type not declared ) : task success {a[0]:.4f} , observation tokens"
      f" {a[4]} , tool errors {a[2]}")

observation form  calls errors steps observation tokens total tokens  task success A  task success B  tokens per answer
raw                  74      8    97                770        10103          0.9583          0.9583                439
truncated 7          74      8    97                428         9160          0.7917          0.7500                482
truncated 4         149    116   156                616        15945          0.2917          0.2917               2278
summarized           74      8    97                401         9229          0.9583          0.9583                401
narrow summary ( type not declared ) : task success 0.7917 , observation tokens 311 , tool errors 8

The baseline is the first row: the raw observation produces 770 observation tokens, spends 10103 total tokens, and holds task success at 0.9583. The second row puts side by side what truncation adds and what it drops. Observation tokens fall to 428, total tokens to 9160, but task success becomes 0.7917. Calls, steps, and tool errors do not change at all; the four tasks that drop are all commercial subscribers, and in all four the invoice comes out exactly half. When the type drops, the tariff tool uses the declared default, that is residential. Truncation produced no error; it produced a silent wrong answer.

The third row is a clear example of the course’s second claim. Squeezing the truncation to four words cuts the second index in the sayac_oku observation as well; the tuketim (consumption) tool returns with a missing field, the chain breaks, and the agent retries the same step. Tool errors rise from 8 to 116, calls from 74 to 149, steps from 97 to 156. The result: observation tokens are still lower than in the raw form (616), but total tokens jump to 15945, one and a half times the raw form. Making the observation cheaper made the loop more expensive.

Declaration and Position

The last row is the number of the summary and it stands alone in the table: task success is 0.9583 in both runs, the same as the raw observation; observation tokens are 401, less than half of the raw; tokens per correct answer fall from 439 to 401. The summary is both the cheapest and the most accurate form, because it reads what to throw away not from position but from declaration.

This advantage is not free, and the last output line writes its cost. When only the tip field is dropped from the summary’s declaration, task success falls to 0.7917, to where truncation dropped it. A summary is only as good as its declaration; whoever knows which field will be needed at the next step must declare it, and if one does not know this, a summary is no safer than truncation.

There is one row in the table that looks reversed: the summary’s observation tokens are lower than those of truncated 7 (401 against 428) but its total tokens are higher (9229 against 9160). The reason is the arithmetic of accumulation. An observation is sent again at every step that comes after it; therefore an early observation is costlier than a late one. The summary keeps the sayac_oku observation at 14 tokens (truncation at 9), and that observation is in the first step of most tasks. In contrast, the summary’s big gain is in belge_ara (search docs; from 19 to 3), and that observation is the last step of single-step tasks, so it is never sent again.

Summary

• A tool result is not written to the state directly: it is turned into an observation text, read back, and sent again in the prompt at every step.
• The raw observation gives 770 observation tokens, 10103 total tokens, and a task success of 0.9583; this is the baseline.
• Truncating to seven words drops the tokens to 9160 but, because it drops the subscriber type, halves four commercial invoices and pulls task success to 0.7917; the tool errors do not rise at all, the wrong answer is silent.
• Truncating to four words breaks the chain: tool errors rise to 116, steps to 156, total tokens to 15945, and task success falls to 0.2917; making the observation cheaper makes the loop more expensive.
• The summary with a declaration keeps the task success of the raw observation while dropping the observation tokens to 401 and the tokens per correct answer to 401; when the type is removed from the same summary, task success falls to 0.7917.
• In an accumulating context an early observation is costlier than a late one: an observation is sent again at every step after it.

Next Step

In these two lessons the tools were defined, called, and their returns formatted. All of them had one thing in common: each tool did a fixed job and returned. A reading is a fixed number of fields, a computation a fixed number of operations, a search a fixed text. That is why how much a call would cost was never asked. In a tool that runs a program the agent wrote itself, this assumption lifts: how many steps it will take, how many cells it will occupy, and how many outputs it will produce are not written in the definition and are seen only while it runs. The next lesson builds such a tool inside a sandbox with resource limits, counts the calls that exceed the declared limit, and writes separately how many tasks stopping it drops.