--- title: 'Tool Definition' source: 'https://academia.sh/en/courses/ai-agents/tool-definition' course: 'AI Agents' language: en updated: '2026-10-21T05:02:27+00:00' license: 'CC BY-SA 4.0' objectives: - "Builds a tool's name, description, schema, and example call as separate definition levels." - 'Compares the valid call rate, the number of tool errors, and task success across the four definition levels.' - 'Tells a call that passes the schema but returns a wrong result apart from a tool error.' - 'Evaluates the token cost of the catalog with the tokens per correct answer measure.' --- # Tool Definition Measuring what a tool's name, description, and schema add to the loop: on 24 fictional tasks the definition is given at four levels, and when only the name is given, 113 of 157 calls come back as tool errors, the valid call rate stays at 0.2803, and task success is 0.3333. Adding the description brings the errors down to 56 and raises task success to 0.6667. Adding the schema drops the call count to 74 and the errors to 8, and the valid rate becomes 0.8919, but task success stalls at 0.7917: the schema gives the field names, not which field carries which value, and two swapped indexes return 99948 instead of 52 in a call that passes the schema. The example call zeroes this share, raises task success to 0.9583, and changes the steps, the calls, and the tool errors not at all; its cost is the catalog growing from 97 to 164 tokens, and tokens per correct answer rising from 1126 to 1212. The previous seven lessons built the agent loop: perception, reasoning, the sequence of action and observation, the separation of the planner and the executor, the step budget, the stopping criterion, the self-critique round, the multi-agent arrangement, and the approval point on a tool with a side effect. One thing was silently assumed throughout: **the tool was ready and correct.** This lesson removes that assumption: what does a tool's **definition** add to the loop? The Building Applications with Model APIs course measured the form of a **single call**: with no catalog given the valid call rate was 0.0500, and with names and parameters given together it was 0.6917. That measurement was about parsing a text and is not repeated; the numbers are taken as input. The call here crosses a function boundary, and what is measured is what the definition adds **across the loop**. - **TM1.** The environment, the tool set, the task set, and the agent loop are **fictional**; no real model endpoint, tool server, or runtime is called. 60 fictional subscribers, 5 fictional documents, 5 tools, and 24 tasks each solved by a known tool sequence. Seed **20260218**, second run **20260219**. - **TM2.** The tool definition is given at four levels: name only; name and description; name, description, and schema; schema and example call. Each level includes the one before it. - **TM3.** The definition determines two things: the multiplier on the share of picking the right tool and the **fill share of each declared field**. The name is a weak hint; the description says what the tool does, and the schema says which fields it asks for. - **TM4.** The schema gives the field **names**; it does not say which value a field carries. Two fields of the same type (`onceki` (previous) and `simdiki` (current)) swap with a share of 0.30; **the example call zeroes this share.** - **TM5.** A **tool error** is an error the tool itself returns: a missing field or a nonexistent tool. Fields outside the schema are filtered at the call boundary; a valid call is one that returns no error. - **TM6.** The catalog stays in the prompt at every step; its cost is the token count of the level. - **TM7.** The baseline is the **name only** level; this is the loop without scaffolding. The step budget is eight at all four levels. - **TM8.** **Task success** is the result the loop leaves being equal to the correct answer; it does not depend on whether the agent stopped on its own. ## The Four Levels of Definition The code below builds the environment, the tool set, the task set, and the agent loop, then adds the definition layer on top of the loop. The call boundary still filters fields outside the schema, but whether a declared field is **filled** now depends on the level of the definition. The tool names in the code are the `read_meter`, `consumption`, `tariff`, `search_docs`, and `write_record` tools of the course's earlier lessons; they are counted in tokens here, so they stay Turkish. ```python SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in str(m).lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- FICTIONAL environment: records of the measurement network # wire words stay Turkish: they are counted in tokens (same rule as the MCP course) # onceki/simdiki/tip/bolge = previous/current/type/region; mesken/isyeri = residential/commercial REGION = ["dogu", "bati", "kuzey", "guney", "merkez"] TYPE = ["mesken", "isyeri"] SUBSCRIBERS = {} for i in range(60): r = rng(SEED + 31 * i) previous = 10000 + int(r() * 80000) SUBSCRIBERS[f"A{i + 1:03d}"] = {"bolge": REGION[int(r() * 5)], "tip": TYPE[int(r() * 2)], "onceki": previous, "simdiki": previous + int(r() * 70)} TIERS, ABOVE = ((10, 4), (25, 7), (40, 11)), 16 COMMERCIAL_FACTOR = 2 DOCUMENTS = {"TR01": "tarife basamaklari 10 25 40 metrekup ve ustu", "TR02": "isyeri abonesinde birim ucret iki katidir", "SY01": "endeks farki tuketimi verir sayac donusu 100000 eklenir", "AR01": "ariza kaydi acildiginda saha ekibi iki gun icinde gider", "KS01": "planli kesinti duyurusu bir hafta once yayimlanir"} CALLS = [0] # tool call counter def _charge(m3, type_): remaining, lower, total = m3, 0, 0 for upper, unit in TIERS: band = min(remaining, upper - lower) if band <= 0: break total += band * unit remaining -= band lower = upper total += remaining * ABOVE return total * (COMMERCIAL_FACTOR if type_ == "isyeri" else 1) def t_read_meter(abone=None, **_): a = SUBSCRIBERS.get(abone) if a is None: return {"hata": "abone yok"} return {"onceki": a["onceki"], "simdiki": a["simdiki"], "tip": a["tip"], "bolge": a["bolge"]} def t_consumption(onceki=None, simdiki=None, **_): if onceki is None or simdiki is None: return {"hata": "eksik alan"} f = simdiki - onceki return {"m3": f + 100000 if f < 0 else f} def t_tariff(m3=None, tip="mesken", **_): if m3 is None: return {"hata": "eksik alan"} return {"tutar": _charge(m3, tip)} def t_document(anahtar=None, **_): if not anahtar: return {"hata": "eksik alan"} q = set(tokenize(anahtar)) best, code = -1, None for k, m in DOCUMENTS.items(): o = len(q & set(tokenize(m))) if o > best: best, code = o, k return {"kod": code, "metin": DOCUMENTS[code]} if best > 0 else {"hata": "bulunamadi"} def t_record(abone=None, sutun=None, deger=None, **_): if not (abone and sutun and deger is not None): return {"hata": "eksik alan"} return {"yazildi": True, "abone": abone, "sutun": sutun, "deger": deger} TOOLS = { "sayac_oku": {"function": t_read_meter, "fields": ("abone",), "side_effect": False}, "tuketim": {"function": t_consumption, "fields": ("onceki", "simdiki"), "side_effect": False}, "tarife": {"function": t_tariff, "fields": ("m3", "tip"), "side_effect": False}, "belge_ara": {"function": t_document, "fields": ("anahtar",), "side_effect": False}, "kayit_yaz": {"function": t_record, "fields": ("abone", "sutun", "deger"), "side_effect": True}, } TOOL_NAMES = list(TOOLS) # ---- FICTIONAL tasks: each one is solved by a known tool sequence TASKS = [] for i in range(24): r = rng(SEED + 9973 * i) sub = f"A{1 + int(r() * 60):03d}" t = pick(r(), [0.30, 0.22, 0.20, 0.28]) if t == 0: TASKS.append({"name": f"G{i + 1:02d}", "kind": "charge", "subscriber": sub, "request": f"{sub} abonesinin bu donem bedeli nedir", "sequence": ("sayac_oku", "tuketim", "tarife")}) elif t == 1: TASKS.append({"name": f"G{i + 1:02d}", "kind": "consumption", "subscriber": sub, "request": f"{sub} abonesinin bu donem tuketimi kac metrekup", "sequence": ("sayac_oku", "tuketim")}) elif t == 2: TASKS.append({"name": f"G{i + 1:02d}", "kind": "document", "subscriber": sub, "request": "sayac donusunde endeks farki nasil hesaplanir", "sequence": ("belge_ara",)}) else: TASKS.append({"name": f"G{i + 1:02d}", "kind": "record", "subscriber": sub, "request": f"{sub} abonesinin bedelini hesapla ve kayda yaz", "sequence": ("sayac_oku", "tuketim", "tarife", "kayit_yaz")}) def correct_answer(g): a = SUBSCRIBERS[g["subscriber"]] f = a["simdiki"] - a["onceki"] m3 = f + 100000 if f < 0 else f if g["kind"] == "consumption": return m3 if g["kind"] == "document": return "SY01" return _charge(m3, a["tip"]) # ---- FICTIONAL agent: at each step it picks a tool or finishes def selection(d): return max(0.34, 0.80 - 0.09 * (d - 1)) # share of picking the right tool, falls with depth DESCRIPTION = { "sayac_oku": "abonenin onceki ve simdiki sayac endeksini dondurur", "tuketim": "iki endeks arasindaki farki metrekup olarak verir", "tarife": "metrekup ve abone tipinden donem bedelini hesaplar", "belge_ara": "anahtar metne en yakin kurgu belgeyi getirir", "kayit_yaz": "bir abonenin bir sutununa deger yazar ; yan etkilidir"} EXAMPLE = {"sayac_oku": "sayac_oku ( abone = A007 )", "tuketim": "tuketim ( onceki = 41200 , simdiki = 41255 )", "tarife": "tarife ( m3 = 55 , tip = mesken )", "belge_ara": "belge_ara ( anahtar = endeks farki )", "kayit_yaz": "kayit_yaz ( abone = A007 , sutun = bedel , deger = 470 )"} # level : name , selection factor , field fill share , swap share DEFINITION = {1: ("name only", 0.62, 0.45, 0.30), 2: ("name + description", 0.86, 0.72, 0.30), 3: ("name + description + schema", 1.00, 1.00, 0.30), 4: ("schema + example call", 1.00, 1.00, 0.00)} def catalog(level, name): s = name if level >= 2: s += " : " + DESCRIPTION[name] if level >= 3: s += " ( " + " , ".join(TOOLS[name]["fields"]) + " )" if level >= 4: s += " ornek : " + EXAMPLE[name] return s CATALOG_TOKENS = {d: sum(len(tokenize(catalog(d, a))) for a in TOOLS) for d in DEFINITION} def call_defined(name, state, level, rd): """The defined variant of the shared call boundary: whether a declared field is filled depends on the level of the definition.""" CALLS[0] += 1 if name not in TOOLS: return {"hata": "arac yok"} _, _, share, swap = DEFINITION[level] kw = {a: state[a] for a in TOOLS[name]["fields"] if a in state and rd() < share} if {"onceki", "simdiki"} <= set(kw) and rd() < swap: kw["onceki"], kw["simdiki"] = kw["simdiki"], kw["onceki"] return TOOLS[name]["function"](**kw) def agent_defined(g, level=4, max_steps=8, seed=SEED): """The shared agent loop; the only differences are that the call passes through the definition layer and the catalog stays in the prompt at every step.""" r = rng(seed ^ trace_id(g["name"])) rd = rng(seed ^ trace_id(g["name"] + "tanim")) factor = DEFINITION[level][1] state = {"abone": g["subscriber"], "anahtar": g["request"], "sutun": "bedel"} trace, step, tok, errors, foreign = [], 0, 0, 0, 0 seq, k = g["sequence"], 0 while step < max_steps: step += 1 tok += 60 + 22 * len(trace) + CATALOG_TOKENS[level] target = seq[k] if k < len(seq) else None if target is None: if r() < 0.88: break choice = TOOL_NAMES[int(r() * len(TOOL_NAMES))] else: choice = target if r() < selection(k + 1) * factor else TOOL_NAMES[int(r() * len(TOOL_NAMES))] if choice == "kayit_yaz": state["deger"] = state.get("sonuc") s = call_defined(choice, state, level, rd) trace.append((choice, "hata" in s)) tok += 30 if "hata" in s: errors += 1 continue foreign += choice not in seq state.update({kk: vv for kk, vv in s.items() if kk != "hata"}) if choice == "tuketim": state["sonuc"] = s["m3"] if choice == "tarife": state["sonuc"] = s["tutar"] if choice == "belge_ara": state["sonuc"] = s["kod"] if choice == target: k += 1 return {"steps": step, "tokens": tok, "errors": errors, "foreign": foreign, "answer": state.get("sonuc")} for d in DEFINITION: print(f"level {d} {DEFINITION[d][0]:<27} catalog {CATALOG_TOKENS[d]:>3} tokens") print(" level 3 line :", catalog(3, "tuketim")) print(" level 4 suffix : ornek :", EXAMPLE["tuketim"]) g = TASKS[2] for d in (3, 4): s = agent_defined(g, d) print(f" {g['name']} {g['kind']:<11} level {d} answer {s['answer']:>6} correct" f" {correct_answer(g)} tool errors {s['errors']}") ``` ``` level 1 name only catalog 10 tokens level 2 name + description catalog 71 tokens level 3 name + description + schema catalog 97 tokens level 4 schema + example call catalog 164 tokens level 3 line : tuketim : iki endeks arasindaki farki metrekup olarak verir ( onceki , simdiki ) level 4 suffix : ornek : tuketim ( onceki = 41200 , simdiki = 41255 ) G03 consumption level 3 answer 99948 correct 52 tool errors 1 G03 consumption level 4 answer 52 correct 52 tool errors 1 ``` The last two lines are the core of this lesson. The same task, the same seed, the same tool sequence, and the **same number of tool errors**. At the third level the agent called the `tuketim` (consumption) tool with the two fields the schema declares, the call passed the schema, the tool returned no error, and the result came out 99948 instead of 52. The schema declares the names `onceki` and `simdiki`; it does not declare which one is smaller. When the fields swap, the difference goes negative, the meter rollover rule kicks in, and the tool returns a consumption close to a hundred thousand. This call is **valid and wrong.** ## The Numbers of the Four Levels The same 24 fictional tasks run at four definition levels; task success is printed separately for the two seeds. ```python def measure(level, seed=SEED): CALLS[0] = 0 ok = steps = tok = errors = foreign = 0 for g in TASKS: s = agent_defined(g, level, seed=seed) ok += s["answer"] == correct_answer(g) steps += s["steps"] tok += s["tokens"] errors += s["errors"] foreign += s["foreign"] return ok / len(TASKS), CALLS[0], errors, foreign, steps, tok print("definition level calls errors valid task success A task success B" " steps tokens tokens per answer") for d in DEFINITION: a, b = measure(d), measure(d, SEED + 1) print(f"{DEFINITION[d][0]:<27} {a[1]:>5} {a[2]:>6} {(a[1] - a[2]) / a[1]:>6.4f}" f" {a[0]:>14.4f} {b[0]:>14.4f} {a[4]:>5} {a[5]:>6} {a[5] / (a[0] * 24):>17.0f}") a = measure(4) print(f"highest level : {a[3]} of {a[1] - a[2]} valid calls call a tool" f" outside the task's sequence") ``` ``` definition level calls errors valid task success A task success B steps tokens tokens per answer name only 157 113 0.2803 0.3333 0.4167 167 27950 3494 name + description 111 56 0.4955 0.6667 0.5833 129 27489 1718 name + description + schema 74 8 0.8919 0.7917 0.7917 97 21387 1126 schema + example call 74 8 0.8919 0.9583 0.9583 97 27886 1212 highest level : 8 of 66 valid calls call a tool outside the task's sequence ``` The baseline is the first row: when only the name is known, **113** of 157 calls come back as tool errors, the valid call rate is **0.2803**, and task success is **0.3333**. Most of the errors are missing fields. The name does not say what the tool wants; the agent cannot know which key to pass from the state it holds. The cost shows in the steps column: although the budget is eight, the total step count is **167**, more than seven per task. A failed call spends a step and does not advance the sequence. When the description is added, the errors fall to **56**, the calls to **111**, and task success becomes **0.6667**. The description does two jobs: it raises the share of picking the right tool and it hints at what information the fields ask for. When the schema is added, the calls fall to **74**, the errors to **8**, the valid rate becomes **0.8919**, and the steps fall to **97**. The remaining eight errors are unrelated to the schema: a wrongly chosen tool fails its own precondition. ## What the Schema Does Not Say The third and fourth rows carry the same calls, the same errors, and the same steps; their task success separates as **0.7917 and 0.9583**. The 0.1666 between them is what the example call adds, and it arrives without adding a single step to the loop. The run-to-run difference is 0.0834 at the first two levels and zero at the last two; 0.1666 is above twice this band and **counts as measured**. The example call supplies the one piece of information the schema cannot: which value the field carries. The cost is in the tokens column. The catalog grows from 97 tokens to **164 tokens**, and because it stays in the prompt at every step, the total tokens rise from **21387 to 27886**. Tokens per correct answer rise from **1126 to 1212**: while the example call raises task success, at this scale it raises the unit cost as well. A scaffolding whose cost is not written down counts as unmeasured; the cost here is sixty-seven tokens. The last line is the outermost limit of the schema. Even at the highest level, **eight of the 66 valid calls** call a tool that is not in the task's sequence. In the Building Applications with Model APIs course the same residue was fourteen wrong tool names; there the call was a text, here it crossed a function boundary and the tool ran correctly. A definition says what **can be called**; it does not say what **should be called**. ## Summary - The tool definition splits into four levels: name only, name and description, schema, and schema and example call. - When only the name is given, 113 of 157 calls come back as tool errors, the valid rate is 0.2803, and task success is 0.3333; a failed call spends a step and raises the total steps to 167. - The description brings the errors down to 56 and the schema to 8; with the schema the calls fall to 74, the steps to 97, and the valid rate becomes 0.8919. - The schema gives the field names and not which value a field carries: two swapped indexes return 99948 instead of 52 in a call that passes the schema. - The example call raises task success from 0.7917 to 0.9583 and changes the steps, the calls, and the errors not at all; its cost is the catalog growing to 164 tokens and the tokens per correct answer rising from 1126 to 1212. - Even at the highest level, eight of the 66 valid calls call a tool outside the task's sequence; being valid is not being correct. ## Next Step In this lesson the tool was called and the dictionary it returned was written to the state as it was; every field was carried to the next step and stayed in the prompt again at every step. This is one reason the tokens column rises above a thousand tokens per task. Carrying the result as it is is not the only option: it can be truncated, it can be summarized, or only the needed field can be kept. Each lowers the tokens, and each risks dropping something. The next lesson measures **in which form** the result is given back to the model and prints task success side by side with the tokens that accumulate across the loop.