Tool and Function Calling

In the previous two lessons the response was a text, and it was read by a person or a simple routing rule. An early label pulled the work earlier, but what did the work was always something else.

When a model output is tied directly to a code execution, the criterion changes. It is not enough for the response to be readable: the function called must exist, the parameter names must match and the values must be acceptable. Multi-step continuation, memory and planning are not here; they are left to the AI Agents course. In this lesson no tool is executed; only the validity of the call is counted.

• AU17. The code is a simulator and the tool registry is fictional too: the measurement network’s support line’s own functions. The requests are fictional, seed 20260218. No tool is run.
• AU18. The tool registry is five functions, each corresponding to one topic; one of them has two parameters. A parameter rule is either a number or a closed set of values.
• AU19. The validator counts four error classes: format (the call could not be parsed at all), fabricated name (a function not in the registry), parameter (missing, extra or invalid value) and no error.
• AU20. The catalog is given at three levels: none, names only, or names and parameters. The simulator changes its tendency to use a registered name according to these three cases; the tendency is fictional, what is measured is the difference between the levels.
• AU21. The concept choice is read only from the last user message. So between the three levels only the form of the call changes; which topic the model sees does not change.
• AU22. The network layer continues. Rate limit, timeout and mid-response cut corrupt the call text; a corrupted call falls into the format or fabricated name class. This is why part of the format error comes from the network, not the prompt and does not shrink with the catalog.
• AU23. The product criterion is the correct tool rate: the response that both passes validation and calls the function corresponding to the topic of the request.
• AU24. The cost is written as input and output tokens; the number of calls is 120 at all three levels.

Tool Registry, Parser and Validator

A function call is the fitting of the text produced by the interface to a registered signature. The code below sets up the registry, the call parser and the validator, then asks the same request at three catalog levels.

# SIMULATOR -- FICTIONAL. No real endpoint is called; the tools are fictional too and
# none of them is executed, only the validity of the call is counted.
SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW, MAX_LENGTH, RATE_EVERY = 240, 24, 37


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def split_pieces(m):
    """FICTIONAL tokenizer; a piece has at most five characters."""
    b = []
    for s in m.lower().replace(",", " , ").split():
        new = True
        while len(s) > 5:
            b.append((s[:5], new))
            s, new = s[5:], False
        b.append((s, new))
    return b


def tokenize(m):
    return [p for p, _ in split_pieces(m)]


def join_pieces(pl):
    s = ""
    for p, new in pl:
        s += (" " if new and s else "") + p
    return s


def hash_text(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


def draw(prompt, seed, salt):
    return rng((seed + salt) ^ hash_text(prompt))()
TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
CLUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "borc": "fatura",
         "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza",
         "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "rande": "sayac",
         "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife",
         "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti"}
TRAPS = {"degis": "tarife", "kayit": "ariza", "ekip": "kesinti", "takip": "tarife"}
CALL_COUNT = [0]

# FICTIONAL tool registry: the measurement network's support line's own non-third-party functions
TOOLS = {"fatura_getir": {"abone": "sayi"},
         "ariza_kaydi_ac": {"abone": "sayi", "konu": TOPICS},
         "sayac_endeks_sor": {"sayac": "sayi"},
         "tarife_bilgisi": {"abone": "sayi"},
         "kesinti_sorgula": {"bolge": "sayi"}}
CORRECT = dict(zip(TOPICS, TOOLS))
FABRICATED = {"fatura": "fatura_sorgula", "ariza": "ariza_bildir", "sayac": "sayac_oku",
              "tarife": "tarife_getir", "kesinti": "kesinti_listele"}
WRONG_KEYS = {"abone": "abone_no", "sayac": "sayac_no", "bolge": "bolge_kodu",
              "konu": "konu_adi"}


def flatten(messages):
    return " ".join(f"{r} : {i}" for r, i in messages)


def call(messages, seed=SEED, temperature=0.6, max_length=MAX_LENGTH):
    """The response is the text of a tool call; no tool is executed."""
    CALL_COUNT[0] += 1
    prompt = flatten(messages)
    toks = tokenize(prompt)
    first = 1 + len(toks) // 40
    empty = {"input": len(toks), "output": 0, "response": "", "first": first}
    if len(toks) > WINDOW:
        return dict(empty, status="context_overflow")
    if CALL_COUNT[0] % RATE_EVERY == 0:
        return dict(empty, status="rate_limit")
    if len(toks) > 90 and draw(prompt, seed, 3) < 0.04:
        return dict(empty, status="timeout")
    last = messages[-1][1]                        # the concept is read only from the last message
    p = {k: 0.15 for k in TOPICS}
    for t in tokenize(last):
        for table, w in ((CLUES, 1.0), (TRAPS, 0.6)):
            if table.get(t) in p:
                p[table[t]] += w
    names = sorted(p)
    from math import exp
    e = [exp(p[a] / temperature) for a in names]
    k = names[pick(draw(last, seed, 5), [x / sum(e) for x in e])]
    has_names, has_params = "arac" in toks, "param" in toks   # is there a catalog in the prompt
    u4 = draw(prompt, seed, 67)
    if u4 < (0.04 if has_names else 0.12):        # free text instead of a call
        text = f"talebiniz {k} konusunda kaydedildi"
    else:
        threshold = 0.15 + (0.73 if has_names else 0.0) + (0.06 if has_params else 0.0)
        name = CORRECT[k] if draw(prompt, seed, 23) < threshold else FABRICATED[k]
        number = next((x for x in prompt.split() if x.isdigit()), "0")
        right_keys = has_params or draw(prompt, seed, 41) < 0.35
        bad_value = draw(prompt, seed, 59) < 0.10
        fields = []
        for a, rule in TOOLS[CORRECT[k]].items():
            key = a if right_keys else WRONG_KEYS[a]
            if rule == "sayi":
                fields.append(f"{key}={'bilinmiyor' if bad_value else number}")
            else:
                fields.append(f"{key}={'acil' if bad_value else k}")
        text = "cagri " + name + " " + " ".join(fields)
    pl, status = split_pieces(text), "done"
    if len(pl) > max_length:
        pl, status = pl[:max_length], "stopped"
    u = draw(prompt, seed, 91)
    if len(pl) > 3 and u < 0.08:
        pl, status = pl[:1 + int(u / 0.08 * (len(pl) - 1))], "cut"
    return {"status": status, "input": len(toks), "output": len(pl),
            "response": join_pieces(pl), "first": first}


def parse_call(y):
    """Extracts the tool call from the response: (name, {parameter: value}) or None."""
    parts = y.split()
    if len(parts) < 2 or parts[0] != "cagri":
        return None
    args = {}
    for x in parts[2:]:
        if "=" not in x:
            return None
        a, _, v = x.partition("=")
        args[a] = v
    return parts[1], args


def validate(c):
    """Four error classes: format, fabricated name, missing/extra parameter, invalid value."""
    if c is None:
        return ["format"]
    name, args = c
    if name not in TOOLS:
        return ["fabricated_name"]
    schema = TOOLS[name]
    h = [f"missing:{a}" for a in schema if a not in args]
    h += [f"extra:{a}" for a in args if a not in schema]
    for a, rule in schema.items():
        if a not in args:
            continue
        if rule == "sayi" and not args[a].isdigit():
            h.append(f"not_a_number:{a}")
        elif rule != "sayi" and args[a] not in rule:
            h.append(f"invalid:{a}")
    return h


PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"),
            ("faturami odedim ama sistemde borc gorunuyor", "fatura"),
            ("fatura ayrintisini nereden gorebilirim", "fatura"),
            ("evde su akmiyor bir ariza var galiba", "ariza"),
            ("baglanti bozuk oldu ve hic calismiyor", "ariza"),
            ("ariza kaydi actim ama ekip gelmedi", "ariza"),
            ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"),
            ("sayac degisimi icin randevu istiyorum", "sayac"),
            ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"),
            ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"),
            ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"),
            ("birim ucret abone tipine gore degisiyor mu", "tarife"),
            ("mahallede kesinti var ne zaman gelecek", "kesinti"),
            ("planli kesinti duyurusu nereden takip edilir", "kesinti"),
            ("aksamdan beri kesinti yasiyoruz", "kesinti")]
REQUESTS = []
for i in range(120):                              # 120 FICTIONAL support requests
    r = rng(SEED + i)
    k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    REQUESTS.append({"id": 4001 + i, "topic": k,
                     "text": [x for x in PATTERNS if x[1] == k][int(r() * 3)][0]})

PLAIN = "destek talebini uygun bir islev cagrisiyla karsila ."
NAMES = PLAIN + " arac adlari : " + " , ".join(TOOLS) + " ."
FULL = NAMES + " parametreler : " + " , ".join(
    f"{a} ( {' '.join(s)} )" for a, s in TOOLS.items()) + " ."

print("tool registry:", " , ".join(f"{a}({' '.join(s)})" for a, s in TOOLS.items()))
for label, prefix in (("no catalog", PLAIN), ("names only", NAMES),
                      ("names and parameters", FULL)):
    x = REQUESTS[0]
    s = call([("sistem", prefix),
              ("kullanici", f"Talep {x['id']} : {x['text']}")])
    errors = validate(parse_call(s["response"]))
    print(f"  {label:<20} input {s['input']:>3}  {s['response']}")
    print(f"                       error {errors if errors else 'none'}")
x = REQUESTS[0]
print("  same request with seed 20260219 ->",
      call([("sistem", FULL), ("kullanici", f"Talep {x['id']} : {x['text']}")],
           SEED + 1)["response"])

tool registry: fatura_getir(abone) , ariza_kaydi_ac(abone konu) , sayac_endeks_sor(sayac) , tarife_bilgisi(abone) , kesinti_sorgula(bolge)
  no catalog           input  32  cagri kesinti_listele bolge_kodu=4001
                       error ['fabricated_name']
  names only           input  57  cagri kesinti_sorgula bolge_kodu=4001
                       error ['missing:bolge', 'extra:bolge_kodu']
  names and parameters input  98  cagri kesinti_sorgula bolge=4001
                       error none
  same request with seed 20260219 -> cagri sayac_endeks_sor sayac=4001

Three lines show the same request, the same topic and three different errors. At the no-catalog level the format is flawless: the response looks like a call, it has a parameter, the value is correct. But there is no function called kesinti_listele (list outages). This is the error that validation catches most readily and the most dangerous one: looked at without the registry, it seems valid.

In the second line the name is now registered, but the parameter name is written bolge_kodu (region code) and the registry expects bolge (region). The validator counts this as two errors: one missing key, one extra key. In the third line both the name and the parameter agree with the registry. The last line follows from the third rule of the course: the same request on a second seed called a different tool, and that call too would have passed validation.

Numbers of the Three Catalog Levels

The same 120 fictional requests are run at three levels and the error classes are counted separately.

def measure(prefix, seed=SEED):
    CALL_COUNT[0] = 0
    tok_in = tok_out = network_failed = valid = fabricated = bad_format = 0
    param_err = right_tool = 0
    for x in REQUESTS:
        s = call([("sistem", prefix),
                  ("kullanici", f"Talep {x['id']} : {x['text']}")], seed)
        tok_in, tok_out = tok_in + s["input"], tok_out + s["output"]
        network_failed += s["status"] != "done"
        parsed = parse_call(s["response"])
        errors = validate(parsed)
        if not errors:
            valid += 1
            right_tool += parsed[0] == CORRECT[x["topic"]]
        elif errors == ["fabricated_name"]:
            fabricated += 1
        elif errors == ["format"]:
            bad_format += 1
        else:
            param_err += 1
    n = len(REQUESTS)
    return (tok_in, tok_out, network_failed, valid / n, fabricated / n, bad_format / n,
            param_err / n, right_tool / n)


LEVELS = [("no catalog", PLAIN), ("names only", NAMES), ("names and parameters", FULL)]
print("level                 input output network    valid A    valid B  fabricated"
      " format   param  correct tool")
for name, prefix in LEVELS:
    a, b = measure(prefix), measure(prefix, SEED + 1)
    print(f"{name:<20} {a[0]:>6} {a[1]:>6} {a[2]:>7}  {a[3]:>9.4f}  {b[3]:>9.4f}"
          f"  {a[4]:>10.4f} {a[5]:>6.4f}  {a[6]:>6.4f}  {a[7]:>12.4f}")

a = measure(FULL)
print(f"\nvalidation caught {round((1 - a[3]) * 120)} calls ;"
      f" of the {round(a[3] * 120)} calls counted valid,"
      f" {round((a[3] - a[7]) * 120)} call the wrong tool")
print("tokens per correct tool :", end=" ")
for name, prefix in LEVELS:
    z = measure(prefix)
    print(f"{name} {(z[0] + z[1]) / (z[7] * 120):.1f}", end="  ")
print()
CALL_COUNT[0] = 0
n = 0
for x in REQUESTS:
    s = call([("sistem", FULL), ("kullanici", f"Talep {x['id']} : {x['text']}")])
    parsed = parse_call(s["response"])
    if parsed and not validate(parsed) and parsed[0] != CORRECT[x["topic"]] and n < 3:
        n += 1
        print(f"  residue : topic {x['topic']:<8} -> {s['response']}")

level                 input output network    valid A    valid B  fabricated format   param  correct tool
no catalog             3694    809      13     0.0500     0.0500      0.6750 0.2083  0.0667        0.0500
names only             6694    825      12     0.2250     0.2833      0.1500 0.0667  0.5583        0.1833
names and parameters  11614    684      25     0.6917     0.6917      0.0500 0.1583  0.1000        0.5750

validation caught 37 calls ; of the 83 calls counted valid, 14 call the wrong tool
tokens per correct tool : no catalog 750.5  names only 341.8  names and parameters 178.2
  residue : topic ariza    -> cagri tarife_bilgisi abone=4004
  residue : topic kesinti  -> cagri ariza_kaydi_ac abone=4011 konu=ariza
  residue : topic tarife   -> cagri fatura_getir abone=4012

The first row is the client without scaffolding and is summarized by a single number: 0.7250 of the responses carry a function name that does not exist. The valid call rate is 0.0500, that is, about one request in twenty produces an executable call. The model fabricates the name of an interface without seeing it, and the names it fabricates look plausible; a human eye cannot tell them apart at once.

The second row is the most instructive row of this lesson. Giving the names lowers the fabricated name share from 0.6750 to 0.1500, but the valid rate rises only to 0.2250. This is because the parameter error jumps from 0.0667 to 0.5583. The error did not disappear, it moved: the model now knocks on the right door but with the wrong key. A measurement that looks at a single number cannot see this shift.

What Validation Caught and What It Did Not

The third row gives both pieces of information at once: valid rate 0.6917, fabricated name 0.0500, parameter error 0.1000. The two runs differ by 0.0583 in the second row, and the steps of 0.1750 and 0.4667 between the levels are many times larger than that; these gains are measured.

The price stands in the input column: from 3694 to 11614 tokens, that is, 3.14 times. In return, the tokens per correct tool fall from 750.5 to 178.2. The catalog is expensive and still cheap, because the tokens paid fall faster than the number of calls thrown away.

The format column does not go down with the levels: 0.2083, then 0.0667, then 0.1583. The reason is in the neighboring column. The number of calls that failed in the network layer is 13, 12 and 25 respectively; the level that sends the longest prompt is the level cut most often, and a cut call text cannot be parsed. Part of the format error comes not from the prompt but from the network, and no catalog reduces it.

The last two lines write the limit of validation. The validator caught 37 calls; of the remaining 83 calls, 14 call a registered but wrong function. In the residue examples, tariff information is requested for a fault request and a fault record for an outage request. These calls pass the schema, are executed and return the wrong result. Validity is not correctness; a schema says what can be called, not what should be called.

Summary

• The tool registry, the parser and the validator with four error classes are written in the lesson with the standard library; no tool is executed.
• When no catalog is given, 0.6750 of the responses fabricate a function name that does not exist and the valid call rate stays at 0.0500.
• Giving only the names lowers the fabricated name to 0.1500 but raises the parameter error to 0.5583: the error does not disappear, it moves.
• When names and parameters are given together, the valid rate is 0.6917 and the correct tool rate is 0.5750; while the input tokens grow from 3694 to 11614, the tokens per correct tool fall from 750.5 to 178.2.
• The format error does not shrink with the catalog, because part of its source is the network: the number of failed calls rises to 25 at the longest prompt.
• Validation catches 37 calls, but of the 83 calls counted as valid, 14 call a registered but wrong function and no schema catches this residue.

Next Step

In this lesson validity was checked afterwards: the response arrived, was parsed, was validated and what did not pass was thrown away. The validator worked well as a counter, but for the 37 calls that did not pass the schema, the only thing it could do was count them. The Prompt Engineering course measured imposing the format by writing it into the prompt and raised the conformance rate to 0.8000. If the interface itself can constrain generation so that it cannot step outside the schema, then there are two differences between imposing with the prompt and imposing with the interface: the conformance rate and the tokens. The next lesson prints these two side by side and counts what enforcement at the interface level adds and what it does not.