Structured Output

The previous two lessons arranged which concept the response chose and where it ended. The response itself was still a sentence: in which word the label appeared, which phrase stood next to it and where a record field could be read from was different every time.

If the output of a support assistant goes not to a person but to a record system, a sentence is unusable. What is needed is a record in which certain fields are filled with certain values. This lesson asks for that record on three levels — free text, a format description written into the prompt and an imposed schema that constrains the generation — and prints the rate of schema-conforming responses of the three side by side with the token cost.

• OC17. The code is a simulator; no real endpoint is called. The tickets, the fields and the value sets are fictional, seed 20260218.
• OC18. The schema is four fields: a ticket number that must be a number, and a topic, a priority and an action, three of which come from closed value sets. The ticket number is written into the prompt, that is, the model does not have to make it up.
• OC19. The schema validator is written inside the lesson; no schema library is used from outside. Four error classes are counted: could not be parsed, missing key, extra key, invalid value.
• OC20. A format description is a tendency, not a guarantee. When the simulator sees a format description in the prompt, it produces a share of the responses in line with the description; the remaining share falls into one of five broken formats. The fifth passes the schema: the fields are in place but the ticket number is a number that is not on record.
• OC21. An imposed schema is a constraint. In this mode the generation cannot leave the schema; the fields are complete and the values come from the closed sets.
• OC22. Every decision in the simulator comes from a single separately salted draw; the decisions are not tied to each other. The topic choice is read only from the ticket text, so only the format changes between the format levels.
• OC23. A recordable correct record is a response that both conforms to the schema and has the correct topic; this is the product criterion of this lesson.
• OC24. The cost is written as input and output tokens; the number of calls is one per ticket at all three levels.

Schema, Parser and Validator

Schema validation was measured in the Web API Design course and in the Cybersecurity curriculum; the procedure is not repeated here. The only thing added is the identity of the producer: the record being validated comes not from a client but from an uncertain component. For this reason the validator is not a security gate but a measuring tool: it counts how many responses fall into which class.

# SIMULATOR -- FICTIONAL. No real model endpoint is called. The schema validation is also
# written inside the lesson; no schema library is used from outside.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(text):                     # FICTIONAL tokenizer, a piece has at most five characters
    out = []
    for piece in text.lower().replace(",", " , ").split():
        while len(piece) > 5:
            out.append(piece[:5])
            piece = piece[5:]
        out.append(piece)
    return out


def trace_id(text):
    h = 2166136261
    for c in text:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
PRIORITIES, ACTIONS = ["dusuk", "orta", "yuksek"], ["kayit", "yonlendirme", "bilgi"]
SCHEMA = {"talep_no": "sayi", "konu": TOPICS, "oncelik": PRIORITIES, "aksiyon": ACTIONS}
PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"),
            ("faturami odedim ama sistemde borc gorunuyor", "fatura"),
            ("fatura ayrintisini nereden gorebilirim", "fatura"),
            ("evde su akmiyor bir ariza var galiba", "ariza"),
            ("baglanti bozuk oldu ve hic calismiyor", "ariza"),
            ("ariza kaydi actim ama ekip gelmedi", "ariza"),
            ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"),
            ("sayac degisimi icin randevu istiyorum", "sayac"),
            ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"),
            ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"),
            ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"),
            ("birim ucret abone tipine gore degisiyor mu", "tarife"),
            ("mahallede kesinti var ne zaman gelecek", "kesinti"),
            ("planli kesinti duyurusu nereden takip edilir", "kesinti"),
            ("aksamdan beri kesinti yasiyoruz", "kesinti")]
TICKETS = []
for i in range(120):                    # 120 FICTIONAL support tickets
    r = rng(SEED + i)
    topic = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    TICKETS.append({"id": 4001 + i, "topic": topic,
                    "text": [x for x in PATTERNS if x[1] == topic][int(r() * 3)][0]})

CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza",
        "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac",
        "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife",
        "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti",
        "mahal": "kesinti"}
TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura",
         "ekip": "kesinti", "takip": "tarife", "giril": "fatura"}   # misleading cues
COUNTER = [0]


def parse_record(text):
    """Key=value pair parser. Returns None for a response that cannot be parsed."""
    if "=" not in text:
        return None
    record = {}
    for part in text.split(";"):
        if "=" not in part:
            return None
        key, _, value = part.partition("=")
        record[key.strip()] = value.strip()
    return record


def validate(record):
    """Schema validation: missing key, extra key, invalid value."""
    if record is None:
        return ["ayristirilamadi"]
    errors = [f"eksik:{k}" for k in SCHEMA if k not in record]
    errors += [f"fazla:{k}" for k in record if k not in SCHEMA]
    for key, rule in SCHEMA.items():
        if key not in record:
            continue
        value = record[key]
        if rule == "sayi" and not value.isdigit():
            errors.append(f"sayi_degil:{key}")
        elif rule != "sayi" and value not in rule:
            errors.append(f"gecersiz:{key}")
    return errors


def draw(prompt, seed, salt):
    """One separately salted draw for each decision; decisions are not tied to each other."""
    return rng((seed + salt) ^ trace_id(prompt))()


def call(prompt, ticket, mode, seed=SEED):
    """mode: free | format | schema. In schema mode the generation cannot leave the schema."""
    COUNTER[0] += 1
    toks = tokenize(prompt)
    p = {k: 0.15 for k in TOPICS}
    for t in tokenize(ticket["text"]):      # the topic is read only from the ticket text
        for table, w in ((CUES, 1.0), (TRAPS, 1.3)):
            if table.get(t) in p:
                p[table[t]] += w
    k = max(sorted(p), key=lambda a: p[a])
    prio = PRIORITIES[pick(draw(prompt, seed, 11), [0.31, 0.44, 0.25])]
    act = ACTIONS[pick(draw(prompt, seed, 29), [0.46, 0.32, 0.22])]
    line = f"talep_no={ticket['id']} ; konu={k} ; oncelik={prio} ; aksiyon={act}"
    if mode == "schema":                                # schema imposition
        return line, k
    if mode == "free":
        return f"talebiniz {k} konusunda kaydedildi ve {act} adimi baslatildi", k
    tendency = 0.62 if "bicim" in toks else 0.0         # format description in the prompt
    if "ornek" in toks:
        tendency += 0.22
    if draw(prompt, seed, 53) < tendency:
        return line, k
    broken = pick(draw(prompt, seed, 71),               # response that does not follow the description
                  [0.22, 0.20, 0.18, 0.16, 0.24])
    if broken == 0:
        return f"talebiniz {k} konusunda kaydedildi ve {act} adimi baslatildi", k
    if broken == 1:
        return f"talep_no={ticket['id']} ; konu={k} ; aksiyon={act}", k
    if broken == 2:
        return f"talep_no={ticket['id']} ; konu={k} ; oncelik=acil ; aksiyon={act}", k
    if broken == 3:
        return f"talep_no=TL{ticket['id']} ; konu={k} ; oncelik={prio} ; aksiyon={act}", k
    return f"talep_no={ticket['id'] + 500} ; konu={k} ; oncelik={prio} ; aksiyon={act}", k


FORMAT_DESCRIPTION = ("Yaniti su bicimde yaz: talep_no=<sayi> ; konu=<fatura|ariza|sayac|"
                      "tarife|kesinti> ; oncelik=<dusuk|orta|yuksek> ; aksiyon=<kayit|"
                      "yonlendirme|bilgi> . ")


def build_prompt(prefix, x):
    return prefix + f"Talep {x['id']} : {x['text']}"


print("schema keys:", " , ".join(SCHEMA))
for mode, prefix in (("free", "Su talebi yanitla . "), ("format", FORMAT_DESCRIPTION),
                     ("schema", "Sema: talep_no ; konu ; oncelik ; aksiyon . ")):
    y, _ = call(build_prompt(prefix, TICKETS[0]), TICKETS[0], mode)
    errors = validate(parse_record(y))
    print(f"mode {mode:<8} {y}")
    print(f"           errors {errors if errors else 'none'}")
print("\nfirst four validation errors in format description mode:")
n = 0
for x in TICKETS:
    y, _ = call(build_prompt(FORMAT_DESCRIPTION, x), x, "format")
    errors = validate(parse_record(y))
    if errors and n < 4:
        n += 1
        print(f"  {errors[0]:<22} {y[:52]}")

schema keys: talep_no , konu , oncelik , aksiyon
mode free     talebiniz kesinti konusunda kaydedildi ve yonlendirme adimi baslatildi
           errors ['ayristirilamadi']
mode format   talep_no=TL4001 ; konu=kesinti ; oncelik=orta ; aksiyon=kayit
           errors ['sayi_degil:talep_no']
mode schema   talep_no=4001 ; konu=kesinti ; oncelik=orta ; aksiyon=bilgi
           errors none

first four validation errors in format description mode:
  sayi_degil:talep_no    talep_no=TL4001 ; konu=kesinti ; oncelik=orta ; aksi
  gecersiz:oncelik       talep_no=4009 ; konu=sayac ; oncelik=acil ; aksiyon=
  ayristirilamadi        talebiniz kesinti konusunda kaydedildi ve yonlendirm
  sayi_degil:talep_no    talep_no=TL4032 ; konu=tarife ; oncelik=orta ; aksiy

The output that the three modes produced on the same ticket shows where the difference lies. The free text has the correct topic in it (kesinti, outage) but no record field; the parser finds nothing. In the format description mode the response is almost correct: all four fields are in place, but two letters have come before the ticket number and the field is no longer a number. The record system rejects this line, even though the content of the response is entirely correct.

The four error examples also say where the breakage is: one falls because a field is not a number, one because it carries a value outside the closed set, one because it could not be parsed at all. All of them are format errors and none is a topic error.

The Conformance and Cost Table of the Three Levels

The same 120 tickets are run with four prompts. The first three ask by writing into the prompt, the last one constrains the generation. The second run is taken with seed 20260219 and the schema-conforming rate is printed separately for it.

EXAMPLE = "Ornek: talep_no=4000 ; konu=ariza ; oncelik=orta ; aksiyon=kayit . "
LEVELS = [("free text", "free", "Su talebi yanitla . "),
          ("format description", "format", FORMAT_DESCRIPTION),
          ("format description + example", "format", FORMAT_DESCRIPTION + EXAMPLE),
          ("imposed schema", "schema", "Sema: talep_no ; konu ; oncelik ; aksiyon . ")]


def read_topic(y, record):
    if record and "konu" in record:
        return record["konu"]
    return next((s for s in y.split() if s in TOPICS), None)


def measure(mode, prefix, seed=SEED):
    COUNTER[0] = 0
    parsed = valid = hits = recordable = n_in = n_out = 0
    tally = {}
    for x in TICKETS:
        prompt = build_prompt(prefix, x)
        y, _ = call(prompt, x, mode, seed)
        record = parse_record(y)
        errors = validate(record)
        parsed += record is not None
        valid += not errors
        hit = read_topic(y, record) == x["topic"]
        hits += hit
        recordable += hit and not errors
        for e in errors:
            tally[e.split(":")[0]] = tally.get(e.split(":")[0], 0) + 1
        n_in += len(tokenize(prompt))
        n_out += len(tokenize(y))
    return (parsed / 120, valid / 120, hits / 120, recordable / 120, n_in, n_out, tally)


print("level                        parseable  valid A  valid B  topic hit  recordable  input  output")
for name, mode, prefix in LEVELS:
    a, b = measure(mode, prefix), measure(mode, prefix, SEED + 1)
    print(f"{name:<28} {a[0]:>9.4f}  {a[1]:>7.4f}  {b[1]:>7.4f}  {a[2]:>9.4f}"
          f"  {a[3]:>10.4f}  {a[4]:>5}  {a[5]:>6}")
for name, mode, prefix in LEVELS[1:3]:
    print(f"{name:<28} error distribution {dict(sorted(measure(mode, prefix)[6].items()))}")

level                        parseable  valid A  valid B  topic hit  recordable  input  output
free text                       0.0000   0.0000   0.0000     0.8833      0.0000   2254    1614
format description              0.9333   0.8000   0.7167     0.8833      0.7000   5614    1789
format description + example    0.9667   0.8833   0.8667     0.8833      0.7750   7654    1789
imposed schema                  1.0000   1.0000   1.0000     0.8833      0.8833   2974    1810
format description           error distribution {'ayristirilamadi': 8, 'eksik': 1, 'gecersiz': 9, 'sayi_degil': 6}
format description + example error distribution {'ayristirilamadi': 4, 'eksik': 5, 'gecersiz': 2, 'sayi_degil': 3}

The first row is a complete loss from the record system’s point of view. The topic hit rate of the free text is 0.8833, that is, about eight of every nine responses state the correct topic. The rate of recordable correct records, however, is 0.0000. The correct information was produced and ignored because it could not be read. This is the clearest row showing why format checking is a criterion and not a decoration.

The second row is the partial gain of writing into the prompt: parseable responses 0.9333, but schema-conforming 0.8000. The 0.1333 in between consists of responses that can be fully parsed but fail validation: in nine the priority is outside the closed set, in six the ticket number is not a number, in one a field is missing. Parseability and validity are not the same thing and cannot be measured with a single number.

The Unmeasurable Gain and the Reverse Side of the Cost

The third row is the one that fits intuition best and sits weakest in the table. Adding an example raises the conformance from 0.8000 to 0.8833; the gain is 0.0833. The two-run columns of the format description row, which is the base of the comparison, need to be looked at: that row gives 0.7167 on the second seed, that is, its two-run difference is also 0.0833. The gain is not larger than the difference. The rule of the course applies directly here: if the difference a prompt change adds does not exceed the two-run difference, it counts as unmeasured. Adding an example may have worked, but this table does not show it.

The cost of the same change is certain, however. The input tokens go from 5614 to 7654: seventeen per ticket, 2040 tokens in total. A measurable cost paid for an unmeasurable gain is the exact counterpart of the second claim of this course.

The fourth row is the opposite of this. The imposed schema raises the conformance to 1.0000; the gain over the base is 0.2000, two and a half times the two-run difference. The cost column is even more surprising: the input tokens stay at 2974. That is a little more than half of the format description and two fifths of the description with the example. This is because the schema is not explained to the prompt at length, it is declared briefly with the field names, and the real work is done in constraining the generation. The cheapest prompt gives the highest conformance. The output tokens, on the other hand, are close to each other at the four levels (between 1614 and 1810), because what changes is not the length of the output but its format.

Finally, the topic hit rate column is fixed at 0.8833 on all four rows, and this is not a coincidence. The imposed schema guarantees the format of the response, not its content. Fourteen tickets are still assigned to the wrong topic, and this wrong topic now stands inside a perfectly formatted record, as a valid value. Validity is not correctness; a record that passes the schema can be wrong. Indeed, in the format description row, thirteen of the records that pass the schema carry a ticket number that is not on record instead of the number written into the prompt, and the validator sees none of them.

Summary

• The schema is four fields and its validator is written inside the lesson with the standard library; the error classes are counted separately.
• In free text the topic hit rate is 0.8833 while the rate of recordable correct records is 0.0000: the correct information is ignored because it cannot be read.
• The format description raises the conformance to 0.8000; parseability (0.9333) and conformance to the schema are not the same thing.
• Adding an example raises the conformance to 0.8833, but the gain is 0.0833 and the two-run difference of the format description row is also 0.0833: the gain counts as unmeasured and 2040 tokens are paid for it.
• The imposed schema raises the conformance to 1.0000 and does so with 2974 input tokens; the cheapest prompt gives the highest conformance.
• The topic hit rate is fixed at 0.8833 at all four levels and thirteen records that pass the schema carry a ticket number that is not on record: the schema guarantees the format, not the content.

Next Step

In this lesson the validator was used only as a counter: it counted how many responses fell, and the fallen responses were thrown away. In a production pipeline throwing away is not an option; a fallen record is either recovered or processed by hand. Moreover, the problem seen in the last paragraph was never addressed: a record that passes the schema can violate a business rule, and a ticket number that is not on record is the clearest example. The next lesson counts the two checks separately and builds a retry loop on top: how many responses do not conform to the schema, how many get caught on a business rule, how many of these the retry recovers and how many extra calls this recovery costs.