Structured Output Mode

The previous lesson checked the validity of a call afterwards: the response arrived, was parsed, was validated and what did not pass was thrown away. The validator was a good counter, but for the calls that did not pass, the only thing it could do was count them.

The Prompt Engineering course measured imposing the format by writing it into the prompt: the share of responses conforming to the schema came out 0.0000 in free text, 0.8000 with a format description and 1.0000 with schema enforcement that constrains generation. Those numbers are the input of this lesson and are not derived again. Two things are added here. First, when the schema can be carried as a request field, it becomes independent of the length of the prompt; as the schema grows, the cost of the two paths diverges. Second, that 1.0000 was measured in a networkless world; in a network operation the same enforcement does not give the same number.

• AU25. The code is a simulator; the requests and the field set are fictional, seed 20260218. The schema validation routine was measured in the Prompt Engineering course and is used here only as a counter.
• AU26. The field pool is twelve fields; the schema is the first four, the first eight or all of them. A field rule is either a number or a closed set of values.
• AU27. Two enforcement paths are measured. In prompt-level enforcement the description of the schema is written into the system message: the field names and the allowed values. In interface-level enforcement the schema is a request field; only the field names are carried and these tokens also enter the count.
• AU28. In interface-level enforcement generation cannot step outside the schema. In prompt-level enforcement each field is produced correctly by an independent draw; in a failed draw the field is skipped, its name is corrupted or its value falls outside the closed set.
• AU29. The network layer continues. Rate limit, timeout and mid-response cut corrupt the record; the network-free conformance column is read only over completed calls and corresponds to the number measured by the Prompt Engineering course.
• AU30. The concept is read only from the last user message, so between the two modes only the enforcement path changes.
• AU31. The product criterion is the correct record: the response that both conforms to the schema and whose topic field equals the topic of the request.
• AU32. The context window and the maximum output length are widened in this lesson for the large records; the thresholds are fictional. The number of calls is 120 in all six setups.

Two Ways of Carrying the Schema

The same schema can travel by two roads: as a sentence inside the prompt, or as a field of the request. In the first, the schema is part of the text the model reads and costs a few tokens per field; in the second only the field names are carried and generation is constrained.

# SIMULATOR -- FICTIONAL. No real endpoint is called; the network layer is in the simulator too.
SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW, MAX_LENGTH, RATE_EVERY = 480, 60, 37


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def split_pieces(m):
    """FICTIONAL tokenizer; a piece has at most five characters."""
    b = []
    for s in m.lower().replace(",", " , ").split():
        new = True
        while len(s) > 5:
            b.append((s[:5], new))
            s, new = s[5:], False
        b.append((s, new))
    return b


def tokenize(m):
    return [p for p, _ in split_pieces(m)]


def join_pieces(pl):
    s = ""
    for p, new in pl:
        s += (" " if new and s else "") + p
    return s


def hash_text(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


def draw(prompt, seed, salt):
    return rng((seed + salt) ^ hash_text(prompt))()


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
CLUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "borc": "fatura",
         "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza",
         "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "rande": "sayac",
         "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife",
         "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti"}
TRAPS = {"degis": "tarife", "kayit": "ariza", "ekip": "kesinti", "takip": "tarife"}
FIELDS = [("talep_no", "sayi"), ("konu", TOPICS),
          ("oncelik", ["dusuk", "orta", "yuksek"]),
          ("aksiyon", ["kayit", "yonlendirme", "bilgi"]), ("abone_no", "sayi"),
          ("kanal", ["telefon", "uygulama", "web"]), ("bolge_no", "sayi"),
          ("durum", ["acik", "beklemede", "kapali"]), ("sayac_no", "sayi"),
          ("ariza_kodu", "sayi"), ("adim_no", "sayi"), ("kayit_no", "sayi")]
CALL_COUNT = [0]


def flatten(messages):
    return " ".join(f"{r} : {i}" for r, i in messages)


def call(messages, schema, at_interface, seed=SEED, temperature=0.6):
    """If the schema is enforced at the interface it is carried as a request field
    and generation cannot step outside it."""
    CALL_COUNT[0] += 1
    prompt = flatten(messages)
    toks = tokenize(prompt)
    if at_interface:
        toks = toks + tokenize(" ".join(schema))  # the schema is carried as a request field
    first = 1 + len(toks) // 40
    empty = {"input": len(toks), "output": 0, "response": "", "first": first}
    if len(toks) > WINDOW:
        return dict(empty, status="context_overflow")
    if CALL_COUNT[0] % RATE_EVERY == 0:
        return dict(empty, status="rate_limit")
    if len(toks) > 90 and draw(prompt, seed, 3) < 0.04:
        return dict(empty, status="timeout")
    last = messages[-1][1]                        # the concept is read only from the last message
    p = {k: 0.15 for k in TOPICS}
    for t in tokenize(last):
        for table, w in ((CLUES, 1.0), (TRAPS, 0.6)):
            if table.get(t) in p:
                p[table[t]] += w
    names = sorted(p)
    from math import exp
    e = [exp(p[a] / temperature) for a in names]
    k = names[pick(draw(last, seed, 5), [x / sum(e) for x in e])]
    number = int(next((x for x in prompt.split() if x.isdigit()), "0"))
    fields = []
    for i, (a, rule) in enumerate(schema.items()):
        value = k if rule == TOPICS else (str(number + i) if rule == "sayi"
                                          else rule[(number + i) % len(rule)])
        if at_interface or draw(prompt, seed, 101 + i) < 0.955:
            fields.append(f"{a}={value}")         # cannot step outside the schema
        else:                                     # three corruptions in prompt-level enforcement
            flaw = pick(draw(prompt, seed, 211 + i), [0.34, 0.33, 0.33])
            if flaw == 1:
                fields.append(f"{a.replace('_', '')}={value}")
            elif flaw == 2:
                fields.append(f"{a}={'x' + value if rule == 'sayi' else 'acil'}")
    pl, status = split_pieces(" ; ".join(fields)), "done"
    if len(pl) > MAX_LENGTH:
        pl, status = pl[:MAX_LENGTH], "stopped"
    u = draw(prompt, seed, 91)                    # the network cuts the response midway
    if len(pl) > 3 and u < 0.08:
        pl, status = pl[:1 + int(u / 0.08 * (len(pl) - 1))], "cut"
    return {"status": status, "input": len(toks), "output": len(pl),
            "response": join_pieces(pl), "first": first}


def validate(y, schema):
    """The schema validation routine was measured in the Prompt Engineering course; here it is a counter."""
    record = {}
    for part in y.split(" ; "):
        a, _, value = part.partition("=")
        record[a.strip()] = value.strip()
    if sorted(record) != sorted(schema):
        return False
    for a, rule in schema.items():
        value = record[a]
        if (not value.isdigit()) if rule == "sayi" else (value not in rule):
            return False
    return True


def read_topic(y, schema):
    for part in y.split(" ; "):
        a, _, value = part.partition("=")
        if a.strip() == "konu" and "konu" in schema:
            return value.strip()
    return None


PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"),
            ("faturami odedim ama sistemde borc gorunuyor", "fatura"),
            ("fatura ayrintisini nereden gorebilirim", "fatura"),
            ("evde su akmiyor bir ariza var galiba", "ariza"),
            ("baglanti bozuk oldu ve hic calismiyor", "ariza"),
            ("ariza kaydi actim ama ekip gelmedi", "ariza"),
            ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"),
            ("sayac degisimi icin randevu istiyorum", "sayac"),
            ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"),
            ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"),
            ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"),
            ("birim ucret abone tipine gore degisiyor mu", "tarife"),
            ("mahallede kesinti var ne zaman gelecek", "kesinti"),
            ("planli kesinti duyurusu nereden takip edilir", "kesinti"),
            ("aksamdan beri kesinti yasiyoruz", "kesinti")]
REQUESTS = []
for i in range(120):                              # 120 FICTIONAL support requests
    r = rng(SEED + i)
    k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    REQUESTS.append({"id": 4001 + i, "topic": k,
                     "text": [x for x in PATTERNS if x[1] == k][int(r() * 3)][0]})


def describe_schema(schema):
    """The form of the schema written into the prompt: field names and allowed values."""
    p = [f"{a}=<{'sayi' if k == 'sayi' else '|'.join(k)}>" for a, k in schema.items()]
    return "Yaniti su bicimde yaz : " + " ; ".join(p) + " ."


def build_prompt(schema, x, at_interface):
    prefix = "" if at_interface else describe_schema(schema)
    return [("sistem", "destek kaydi cikaran bir yardimcisin . " + prefix),
            ("kullanici", f"Talep {x['id']} : {x['text']}")]


SCHEMA4 = dict(FIELDS[:4])
for at_interface in (False, True):
    s = call(build_prompt(SCHEMA4, REQUESTS[0], at_interface), SCHEMA4, at_interface)
    path = "interface" if at_interface else "prompt"
    print(f"{path:<9} input {s['input']:>3}  output {s['output']:>2}  "
          f"conformance {validate(s['response'], SCHEMA4)}  {s['response']}")
print("description of the four-field schema:", len(tokenize(describe_schema(SCHEMA4))),
      "tokens , as a request field:", len(tokenize(" ".join(SCHEMA4))))
print("twelve-field:", len(tokenize(describe_schema(dict(FIELDS)))), "tokens ,",
      len(tokenize(" ".join(dict(FIELDS)))), "tokens")
print("same request with seed 20260219 ->",
      call(build_prompt(SCHEMA4, REQUESTS[0], False), SCHEMA4, False, SEED + 1)["response"])

prompt    input  65  output 15  conformance True  talep_no=4001 ; konu=kesinti ; oncelik=orta ; aksiyon=bilgi
interface input  37  output 15  conformance True  talep_no=4001 ; konu=kesinti ; oncelik=orta ; aksiyon=bilgi
description of the four-field schema: 35 tokens , as a request field: 7
twelve-field: 74 tokens , 21 tokens
same request with seed 20260219 -> talep_no=4001 ; konu=sayac ; oncelik=orta ; aksiyon=bilgi

The two paths produced the same record on the same request and both passed validation; the difference is in the input column. The description of the four-field schema is 35 tokens, the same schema carried as a request field is 7 tokens. The gap continues as the schema grows: 74 against 21 at twelve fields. The description has to carry the field names, the allowed values and an instruction sentence; the request field carries only the names and the interface already knows the rule.

The last line follows from the third rule of the course: the same request on a second seed wrote a different value into the topic field, and that record too would have passed the schema.

Conformance and Cost Table of the Six Setups

The same 120 fictional requests are run on two enforcement paths and at three schema sizes.

def measure(n, at_interface, seed=SEED):
    CALL_COUNT[0] = 0
    schema = dict(FIELDS[:n])
    tok_in = tok_out = net_fail = conforming = done = done_ok = good = 0
    for x in REQUESTS:
        s = call(build_prompt(schema, x, at_interface), schema, at_interface, seed)
        tok_in, tok_out = tok_in + s["input"], tok_out + s["output"]
        ok = validate(s["response"], schema)
        net_fail += s["status"] != "done"
        conforming += ok
        if s["status"] == "done":
            done, done_ok = done + 1, done_ok + ok
        good += ok and read_topic(s["response"], schema) == x["topic"]
    return tok_in, tok_out, net_fail, conforming / 120, done_ok / done, good / 120


print("mode      fields  input output network  conformance A  conformance B"
      "  network-free conformance  correct record")
for at_interface in (False, True):
    for n in (4, 8, 12):
        a, b = measure(n, at_interface), measure(n, at_interface, SEED + 1)
        print(f"{'interface' if at_interface else 'prompt':<9} {n:>6} {a[0]:>6} {a[1]:>6}"
              f" {a[2]:>7}  {a[3]:>13.4f}  {b[3]:>13.4f}  {a[4]:>24.4f}  {a[5]:>14.4f}")

for n in (4, 12):
    i, y = measure(n, False), measure(n, True)
    print(f"\n{n} fields : interface saves {i[0] - y[0]} input tokens ,"
          f" conformance gap {y[3] - i[3]:.4f} ,"
          f" tokens per correct record {(i[0] + i[1]) / (i[5] * 120):.1f}"
          f" -> {(y[0] + y[1]) / (y[5] * 120):.1f}")

mode      fields  input output network  conformance A  conformance B  network-free conformance  correct record
prompt         4   7654   1658      11         0.7583         0.8167                    0.8349          0.6167
prompt         8  10294   3361      14         0.7250         0.7333                    0.8208          0.6000
prompt        12  12334   4894      16         0.6083         0.5583                    0.7019          0.5000
interface      4   4294   1688      12         0.9000         0.9167                    1.0000          0.7583
interface      8   5014   3417      12         0.9000         0.9167                    1.0000          0.7583
interface     12   5974   5219      12         0.9000         0.9167                    1.0000          0.7583

4 fields : interface saves 3360 input tokens , conformance gap 0.1417 , tokens per correct record 125.8 -> 65.7

12 fields : interface saves 6360 input tokens , conformance gap 0.2917 , tokens per correct record 287.1 -> 123.0

The prompt-level enforcement rows go down as the schema grows: conformance 0.7583, then 0.7250, then 0.6083. The cause is structural. If every field is tied to an independent draw, the probability that all the fields are correct at once falls exponentially with the number of fields; a small risk paid at four fields becomes an unaffordable one at twelve. The interface-level enforcement rows stay at 0.9000 in all three sizes. The schema grew, the conformance did not change.

The input column shows the same divergence in cost. Prompt-level enforcement goes from 7654 to 12334 tokens while interface-level enforcement goes from 4294 to 5974; the saving is 3360 tokens at four fields and 6360 tokens at twelve fields. The output column grows in both modes (from 1688 to 5219), because the record really gets longer. At twelve fields the output of interface-level enforcement is more than that of prompt-level enforcement: 5219 against 4894. The reason is that corrupted fields are skipped; a record with missing fields is shorter.

Network-Free Conformance and the Share of the Network

The most important column of the table is network-free conformance. In interface-level enforcement this column is exactly 1.0000 in all three rows. This is the number measured by the Prompt Engineering course and it comes out here unchanged: if generation cannot step outside the schema, every completed response conforms to the schema.

The conformance column of the same rows is 0.9000. The gap between it and 1.0000 is the share not of the model but of the network: 12 of the 120 calls are caught in a rate limit, a timeout or a mid-response cut, and a half record does not pass the schema. The rule of the course becomes directly visible here. Measured as a function call, the conformance of schema enforcement is 1.0000; measured as a network operation, it is 0.9000. A call whose failure is not written down counts as unmeasured, and the difference between the two is exactly that unwritten failure.

The two runs differ by at most 0.0583 (prompt-level enforcement, four fields). The gain of the interface is 0.1417 at four fields and 0.2917 at twelve fields; both are above this difference and are measured. The tokens per correct record fall from 125.8 to 65.7 at four fields and from 287.1 to 123.0 at twelve fields.

The last column carries a warning. In interface-level enforcement the correct record rate is fixed at 0.7583 in all three sizes; the gap between it and the conformance 0.9000 is records that pass the schema but have the wrong topic. Adding fields tripled the output tokens and did not change the correct record rate at all. A guarantee of the format is not a guarantee of the content; this distinction was measured in the Prompt Engineering course and here it stands once more, at the interface level, in the same place.

Summary

• The description of the same schema written into the prompt costs 35 tokens at four fields and 74 at twelve fields; its form carried as a request field is 7 and 21 tokens.
• The conformance of prompt-level enforcement falls from 0.7583 to 0.6083 as the schema grows, because each field is an independent risk; interface-level enforcement stays at 0.9000 in all three sizes.
• The input tokens go from 7654 to 12334 with the prompt and from 4294 to 5974 with the interface; the saving is 3360 tokens at four fields and 6360 at twelve fields.
• The network-free conformance of interface-level enforcement is exactly 1.0000; when the network is counted it falls to 0.9000, and the gap between the two is the share of the network, not the model.
• The tokens per correct record fall from 125.8 to 65.7 at four fields and from 287.1 to 123.0 at twelve fields; these gains are measured because the difference between the two runs is at most 0.0583.
• Adding fields grows the output tokens from 1688 to 5219 but leaves the correct record rate fixed at 0.7583: a format guarantee is not a content guarantee.

Next Step

In four lessons the four abilities of the interface were measured and the cost of each was written in tokens: carrying the conversation state, receiving the response chunk by chunk, tying the output to a call and fitting the output to a schema. The four shared an assumption that was never questioned: the input was always text. The request was a sentence, the state was a transcript, the schema was a list of fields and all of them went through the same tokenizer.

When a photograph of a meter, a voice message or a short video arrives next to a support request, none of these numbers stays as it is. If the same information can also be given as text, how many tokens does each cost and what happens to accuracy when the form changes? The next lesson makes the form of the input a variable and counts its cost on the same scale.