--- title: 'Tool and Function Calling' source: 'https://academia.sh/en/courses/model-apis/tool-and-function-calling' course: 'Building Applications with Model APIs' language: en updated: '2026-10-21T05:02:32+00:00' license: 'CC BY-SA 4.0' objectives: - 'Writes a tool registry, a call parser and a validator with four error classes using the standard library.' - 'Compares the valid call rate, the fabricated tool name share and the parameter error at the three levels of the tool catalog.' - 'Counts separately the number of calls validation catches and the residue it cannot catch.' - 'Evaluates the token cost of providing a catalog with the tokens per correct tool criterion.' --- # Tool and Function Calling Measuring how a model output is tied to a call that will be executed: on 120 fictional requests, when the tool catalog is not written into the prompt the valid call rate is 0.0500 and 0.6750 of the responses fabricate a tool name that does not exist. When only the names are given, the fabricated name share falls to 0.1500 but the parameter error jumps from 0.0667 to 0.5583: the error does not disappear, it moves. When names and parameters are given together, the valid rate is 0.6917 and the rate of calling the correct tool is 0.5750, and the price is input growing from 3694 to 11614 tokens. Still, the tokens per correct tool fall from 750.5 to 178.2. Validation catches 37 calls, but of the 83 calls counted as valid, 14 call a registered but wrong tool, and no schema catches this residue. The format error does not shrink with the catalog, because its source is the network, not the prompt. In the previous two lessons the response was a text, and it was read by a person or a simple routing rule. An early label pulled the work earlier, but what did the work was always something else. When a model output is tied directly to a **code execution**, the criterion changes. It is not enough for the response to be readable: the function called must **exist**, the parameter names must **match** and the values must be **acceptable**. Multi-step continuation, memory and planning are **not** here; they are left to the AI Agents course. In this lesson no tool is **executed**; only the validity of the call is counted. - **AU17.** The code is a **simulator** and **the tool registry is fictional too**: the measurement network's support line's own functions. The requests are fictional, seed **20260218**. No tool is run. - **AU18. The tool registry is five functions**, each corresponding to one topic; one of them has two parameters. A parameter rule is either a **number** or a closed set of values. - **AU19. The validator counts four error classes:** format (the call could not be parsed at all), **fabricated name** (a function not in the registry), parameter (missing, extra or invalid value) and no error. - **AU20. The catalog is given at three levels:** none, **names** only, or **names and parameters**. The simulator changes its tendency to use a registered name according to these three cases; the tendency is fictional, what is measured is the difference between the levels. - **AU21. The concept choice is read only from the last user message.** So between the three levels only the form of the call changes; which topic the model sees does not change. - **AU22. The network layer continues.** Rate limit, timeout and mid-response cut corrupt the call text; a corrupted call falls into the format or fabricated name class. This is why **part of the format error comes from the network, not the prompt** and does not shrink with the catalog. - **AU23. The product criterion is the correct tool rate:** the response that both passes validation and calls the function corresponding to the topic of the request. - **AU24.** The cost is written as input and output tokens; the number of calls is **120** at all three levels. ## Tool Registry, Parser and Validator A function call is the fitting of the text produced by the interface to a **registered signature**. The code below sets up the registry, the call parser and the validator, then asks the same request at three catalog levels. ```python # SIMULATOR -- FICTIONAL. No real endpoint is called; the tools are fictional too and # none of them is executed, only the validity of the call is counted. SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_LENGTH, RATE_EVERY = 240, 24, 37 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def split_pieces(m): """FICTIONAL tokenizer; a piece has at most five characters.""" b = [] for s in m.lower().replace(",", " , ").split(): new = True while len(s) > 5: b.append((s[:5], new)) s, new = s[5:], False b.append((s, new)) return b def tokenize(m): return [p for p, _ in split_pieces(m)] def join_pieces(pl): s = "" for p, new in pl: s += (" " if new and s else "") + p return s def hash_text(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h def draw(prompt, seed, salt): return rng((seed + salt) ^ hash_text(prompt))() TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] CLUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "borc": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "rande": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti"} TRAPS = {"degis": "tarife", "kayit": "ariza", "ekip": "kesinti", "takip": "tarife"} CALL_COUNT = [0] # FICTIONAL tool registry: the measurement network's support line's own non-third-party functions TOOLS = {"fatura_getir": {"abone": "sayi"}, "ariza_kaydi_ac": {"abone": "sayi", "konu": TOPICS}, "sayac_endeks_sor": {"sayac": "sayi"}, "tarife_bilgisi": {"abone": "sayi"}, "kesinti_sorgula": {"bolge": "sayi"}} CORRECT = dict(zip(TOPICS, TOOLS)) FABRICATED = {"fatura": "fatura_sorgula", "ariza": "ariza_bildir", "sayac": "sayac_oku", "tarife": "tarife_getir", "kesinti": "kesinti_listele"} WRONG_KEYS = {"abone": "abone_no", "sayac": "sayac_no", "bolge": "bolge_kodu", "konu": "konu_adi"} def flatten(messages): return " ".join(f"{r} : {i}" for r, i in messages) def call(messages, seed=SEED, temperature=0.6, max_length=MAX_LENGTH): """The response is the text of a tool call; no tool is executed.""" CALL_COUNT[0] += 1 prompt = flatten(messages) toks = tokenize(prompt) first = 1 + len(toks) // 40 empty = {"input": len(toks), "output": 0, "response": "", "first": first} if len(toks) > WINDOW: return dict(empty, status="context_overflow") if CALL_COUNT[0] % RATE_EVERY == 0: return dict(empty, status="rate_limit") if len(toks) > 90 and draw(prompt, seed, 3) < 0.04: return dict(empty, status="timeout") last = messages[-1][1] # the concept is read only from the last message p = {k: 0.15 for k in TOPICS} for t in tokenize(last): for table, w in ((CLUES, 1.0), (TRAPS, 0.6)): if table.get(t) in p: p[table[t]] += w names = sorted(p) from math import exp e = [exp(p[a] / temperature) for a in names] k = names[pick(draw(last, seed, 5), [x / sum(e) for x in e])] has_names, has_params = "arac" in toks, "param" in toks # is there a catalog in the prompt u4 = draw(prompt, seed, 67) if u4 < (0.04 if has_names else 0.12): # free text instead of a call text = f"talebiniz {k} konusunda kaydedildi" else: threshold = 0.15 + (0.73 if has_names else 0.0) + (0.06 if has_params else 0.0) name = CORRECT[k] if draw(prompt, seed, 23) < threshold else FABRICATED[k] number = next((x for x in prompt.split() if x.isdigit()), "0") right_keys = has_params or draw(prompt, seed, 41) < 0.35 bad_value = draw(prompt, seed, 59) < 0.10 fields = [] for a, rule in TOOLS[CORRECT[k]].items(): key = a if right_keys else WRONG_KEYS[a] if rule == "sayi": fields.append(f"{key}={'bilinmiyor' if bad_value else number}") else: fields.append(f"{key}={'acil' if bad_value else k}") text = "cagri " + name + " " + " ".join(fields) pl, status = split_pieces(text), "done" if len(pl) > max_length: pl, status = pl[:max_length], "stopped" u = draw(prompt, seed, 91) if len(pl) > 3 and u < 0.08: pl, status = pl[:1 + int(u / 0.08 * (len(pl) - 1))], "cut" return {"status": status, "input": len(toks), "output": len(pl), "response": join_pieces(pl), "first": first} def parse_call(y): """Extracts the tool call from the response: (name, {parameter: value}) or None.""" parts = y.split() if len(parts) < 2 or parts[0] != "cagri": return None args = {} for x in parts[2:]: if "=" not in x: return None a, _, v = x.partition("=") args[a] = v return parts[1], args def validate(c): """Four error classes: format, fabricated name, missing/extra parameter, invalid value.""" if c is None: return ["format"] name, args = c if name not in TOOLS: return ["fabricated_name"] schema = TOOLS[name] h = [f"missing:{a}" for a in schema if a not in args] h += [f"extra:{a}" for a in args if a not in schema] for a, rule in schema.items(): if a not in args: continue if rule == "sayi" and not args[a].isdigit(): h.append(f"not_a_number:{a}") elif rule != "sayi" and args[a] not in rule: h.append(f"invalid:{a}") return h PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"), ("faturami odedim ama sistemde borc gorunuyor", "fatura"), ("fatura ayrintisini nereden gorebilirim", "fatura"), ("evde su akmiyor bir ariza var galiba", "ariza"), ("baglanti bozuk oldu ve hic calismiyor", "ariza"), ("ariza kaydi actim ama ekip gelmedi", "ariza"), ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"), ("sayac degisimi icin randevu istiyorum", "sayac"), ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"), ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"), ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"), ("birim ucret abone tipine gore degisiyor mu", "tarife"), ("mahallede kesinti var ne zaman gelecek", "kesinti"), ("planli kesinti duyurusu nereden takip edilir", "kesinti"), ("aksamdan beri kesinti yasiyoruz", "kesinti")] REQUESTS = [] for i in range(120): # 120 FICTIONAL support requests r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] REQUESTS.append({"id": 4001 + i, "topic": k, "text": [x for x in PATTERNS if x[1] == k][int(r() * 3)][0]}) PLAIN = "destek talebini uygun bir islev cagrisiyla karsila ." NAMES = PLAIN + " arac adlari : " + " , ".join(TOOLS) + " ." FULL = NAMES + " parametreler : " + " , ".join( f"{a} ( {' '.join(s)} )" for a, s in TOOLS.items()) + " ." print("tool registry:", " , ".join(f"{a}({' '.join(s)})" for a, s in TOOLS.items())) for label, prefix in (("no catalog", PLAIN), ("names only", NAMES), ("names and parameters", FULL)): x = REQUESTS[0] s = call([("sistem", prefix), ("kullanici", f"Talep {x['id']} : {x['text']}")]) errors = validate(parse_call(s["response"])) print(f" {label:<20} input {s['input']:>3} {s['response']}") print(f" error {errors if errors else 'none'}") x = REQUESTS[0] print(" same request with seed 20260219 ->", call([("sistem", FULL), ("kullanici", f"Talep {x['id']} : {x['text']}")], SEED + 1)["response"]) ``` ``` tool registry: fatura_getir(abone) , ariza_kaydi_ac(abone konu) , sayac_endeks_sor(sayac) , tarife_bilgisi(abone) , kesinti_sorgula(bolge) no catalog input 32 cagri kesinti_listele bolge_kodu=4001 error ['fabricated_name'] names only input 57 cagri kesinti_sorgula bolge_kodu=4001 error ['missing:bolge', 'extra:bolge_kodu'] names and parameters input 98 cagri kesinti_sorgula bolge=4001 error none same request with seed 20260219 -> cagri sayac_endeks_sor sayac=4001 ``` Three lines show the same request, the same topic and three different errors. At the no-catalog level the format is flawless: the response looks like a call, it has a parameter, the value is correct. But there is **no** function called `kesinti_listele` (list outages). This is the error that validation catches most readily and the most dangerous one: looked at without the registry, it seems valid. In the second line the name is now registered, but the parameter name is written `bolge_kodu` (region code) and the registry expects `bolge` (region). The validator counts this as two errors: one missing key, one extra key. In the third line both the name and the parameter agree with the registry. The last line follows from the third rule of the course: the same request on a second seed called **a different tool**, and that call too would have passed validation. ## Numbers of the Three Catalog Levels The same 120 fictional requests are run at three levels and the error classes are counted separately. ```python def measure(prefix, seed=SEED): CALL_COUNT[0] = 0 tok_in = tok_out = network_failed = valid = fabricated = bad_format = 0 param_err = right_tool = 0 for x in REQUESTS: s = call([("sistem", prefix), ("kullanici", f"Talep {x['id']} : {x['text']}")], seed) tok_in, tok_out = tok_in + s["input"], tok_out + s["output"] network_failed += s["status"] != "done" parsed = parse_call(s["response"]) errors = validate(parsed) if not errors: valid += 1 right_tool += parsed[0] == CORRECT[x["topic"]] elif errors == ["fabricated_name"]: fabricated += 1 elif errors == ["format"]: bad_format += 1 else: param_err += 1 n = len(REQUESTS) return (tok_in, tok_out, network_failed, valid / n, fabricated / n, bad_format / n, param_err / n, right_tool / n) LEVELS = [("no catalog", PLAIN), ("names only", NAMES), ("names and parameters", FULL)] print("level input output network valid A valid B fabricated" " format param correct tool") for name, prefix in LEVELS: a, b = measure(prefix), measure(prefix, SEED + 1) print(f"{name:<20} {a[0]:>6} {a[1]:>6} {a[2]:>7} {a[3]:>9.4f} {b[3]:>9.4f}" f" {a[4]:>10.4f} {a[5]:>6.4f} {a[6]:>6.4f} {a[7]:>12.4f}") a = measure(FULL) print(f"\nvalidation caught {round((1 - a[3]) * 120)} calls ;" f" of the {round(a[3] * 120)} calls counted valid," f" {round((a[3] - a[7]) * 120)} call the wrong tool") print("tokens per correct tool :", end=" ") for name, prefix in LEVELS: z = measure(prefix) print(f"{name} {(z[0] + z[1]) / (z[7] * 120):.1f}", end=" ") print() CALL_COUNT[0] = 0 n = 0 for x in REQUESTS: s = call([("sistem", FULL), ("kullanici", f"Talep {x['id']} : {x['text']}")]) parsed = parse_call(s["response"]) if parsed and not validate(parsed) and parsed[0] != CORRECT[x["topic"]] and n < 3: n += 1 print(f" residue : topic {x['topic']:<8} -> {s['response']}") ``` ``` level input output network valid A valid B fabricated format param correct tool no catalog 3694 809 13 0.0500 0.0500 0.6750 0.2083 0.0667 0.0500 names only 6694 825 12 0.2250 0.2833 0.1500 0.0667 0.5583 0.1833 names and parameters 11614 684 25 0.6917 0.6917 0.0500 0.1583 0.1000 0.5750 validation caught 37 calls ; of the 83 calls counted valid, 14 call the wrong tool tokens per correct tool : no catalog 750.5 names only 341.8 names and parameters 178.2 residue : topic ariza -> cagri tarife_bilgisi abone=4004 residue : topic kesinti -> cagri ariza_kaydi_ac abone=4011 konu=ariza residue : topic tarife -> cagri fatura_getir abone=4012 ``` The first row is the client without scaffolding and is summarized by a single number: **0.7250** of the responses carry a function name that does not exist. The valid call rate is **0.0500**, that is, about one request in twenty produces an executable call. The model **fabricates** the name of an interface without seeing it, and the names it fabricates look plausible; a human eye cannot tell them apart at once. The second row is the most instructive row of this lesson. Giving the names lowers the fabricated name share **from 0.6750 to 0.1500**, but the valid rate rises only to **0.2250**. This is because the parameter error jumps **from 0.0667 to 0.5583**. The error did not disappear, it **moved**: the model now knocks on the right door but with the wrong key. A measurement that looks at a single number cannot see this shift. ## What Validation Caught and What It Did Not The third row gives both pieces of information at once: valid rate **0.6917**, fabricated name **0.0500**, parameter error **0.1000**. The two runs differ by 0.0583 in the second row, and the steps of 0.1750 and 0.4667 between the levels are many times larger than that; these gains are **measured**. The price stands in the input column: **from 3694 to 11614 tokens**, that is, **3.14 times**. In return, the tokens per correct tool fall **from 750.5 to 178.2**. The catalog is expensive and still cheap, because the tokens paid fall faster than the number of calls thrown away. The format column does not go down with the levels: 0.2083, then 0.0667, then 0.1583. The reason is in the neighboring column. The number of calls that failed in the network layer is 13, 12 and **25** respectively; the level that sends the longest prompt is the level cut most often, and a cut call text cannot be parsed. Part of the format error comes not from the prompt but from the **network**, and no catalog reduces it. The last two lines write the limit of validation. The validator caught **37 calls**; of the remaining 83 calls, **14** call a registered but **wrong** function. In the residue examples, tariff information is requested for a fault request and a fault record for an outage request. These calls pass the schema, are executed and return the **wrong result**. Validity is not correctness; a schema says what can be called, not what should be called. ## Summary - The tool registry, the parser and the validator with four error classes are written in the lesson with the standard library; no tool is executed. - When no catalog is given, 0.6750 of the responses fabricate a function name that does not exist and the valid call rate stays at 0.0500. - Giving only the names lowers the fabricated name to 0.1500 but raises the parameter error to 0.5583: the error does not disappear, it moves. - When names and parameters are given together, the valid rate is 0.6917 and the correct tool rate is 0.5750; while the input tokens grow from 3694 to 11614, the tokens per correct tool fall from 750.5 to 178.2. - The format error does not shrink with the catalog, because part of its source is the network: the number of failed calls rises to 25 at the longest prompt. - Validation catches 37 calls, but of the 83 calls counted as valid, 14 call a registered but wrong function and no schema catches this residue. ## Next Step In this lesson validity was checked **afterwards**: the response arrived, was parsed, was validated and what did not pass was thrown away. The validator worked well as a counter, but for the 37 calls that did not pass the schema, the only thing it could do was count them. The Prompt Engineering course measured imposing the format **by writing it into the prompt** and raised the conformance rate to 0.8000. If the interface itself can constrain generation so that it **cannot step outside** the schema, then there are two differences between imposing with the prompt and imposing with the interface: the conformance rate and the tokens. The next lesson prints these two side by side and counts what enforcement at the interface level adds and what it does not.