--- title: 'Structured Output' source: 'https://academia.sh/en/courses/prompt-engineering/structured-output' course: 'Prompt Engineering' language: en updated: '2026-10-21T05:02:33+00:00' license: 'CC BY-SA 4.0' objectives: - 'Writes a schema made of key-value pairs and its validator with the standard library.' - 'Compares the rate of schema-conforming responses of free text, a format description and an imposed schema on the same set.' - 'Calculates the input and output token cost of each level and tests the conformance gain against the two-run difference.' - 'Shows with numbers why format conformance does not guarantee content accuracy.' --- # Structured Output Measuring free text, a format description written into the prompt and an imposed schema side by side on the same 120 fictional tickets: the rate of schema-conforming responses is 0.0000 in free text, 0.8000 with the format description, 0.8833 when an example is added and 1.0000 with the imposed schema. The two-run difference of the format description row is 0.0833 and, since the gain from adding an example is also 0.0833, that gain counts as unmeasured, and it adds 2040 tokens. Topic hit rate stays fixed at 0.8833 at all four levels. Input tokens rise from 2254 to 5614 with the format description and to 7654 with the example, while the imposed schema stays at 2974; that is, the cheapest prompt gives the highest conformance. The previous two lessons arranged **which concept** the response chose and **where it ended**. The response itself was still a sentence: in which word the label appeared, which phrase stood next to it and where a record field could be read from was different every time. If the output of a support assistant goes not to a person but to a **record system**, a sentence is unusable. What is needed is a record in which certain fields are filled with certain values. This lesson asks for that record on three levels — free text, a **format description** written into the prompt and an **imposed schema** that constrains the generation — and prints the **rate of schema-conforming responses** of the three side by side with the token cost. - **OC17.** The code is a **simulator**; no real endpoint is called. The tickets, the fields and the value sets are **fictional**, seed **20260218**. - **OC18. The schema is four fields:** a ticket number that must be a number, and a topic, a priority and an action, three of which come from closed value sets. The ticket number is written into the prompt, that is, the model does not have to make it up. - **OC19.** The schema validator is **written inside the lesson**; no schema library is used from outside. Four error classes are counted: could not be parsed, missing key, extra key, invalid value. - **OC20. A format description is a tendency, not a guarantee.** When the simulator sees a format description in the prompt, it produces a share of the responses in line with the description; the remaining share falls into one of five broken formats. The fifth **passes the schema**: the fields are in place but the ticket number is a number that is not on record. - **OC21. An imposed schema is a constraint.** In this mode the generation **cannot** leave the schema; the fields are complete and the values come from the closed sets. - **OC22.** Every decision in the simulator comes from **a single separately salted draw**; the decisions are not tied to each other. The topic choice is read only from the **ticket text**, so only the format changes between the format levels. - **OC23. A recordable correct record** is a response that both conforms to the schema and has the correct topic; this is the product criterion of this lesson. - **OC24.** The cost is written as input and output tokens; the number of calls is one per ticket at all three levels. ## Schema, Parser and Validator Schema validation was measured in the Web API Design course and in the Cybersecurity curriculum; the procedure is not repeated here. The only thing added is the identity of the producer: the record being validated comes not from a client but from an **uncertain component**. For this reason the validator is not a security gate but a **measuring tool**: it counts how many responses fall into which class. ```python # SIMULATOR -- FICTIONAL. No real model endpoint is called. The schema validation is also # written inside the lesson; no schema library is used from outside. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): # FICTIONAL tokenizer, a piece has at most five characters out = [] for piece in text.lower().replace(",", " , ").split(): while len(piece) > 5: out.append(piece[:5]) piece = piece[5:] out.append(piece) return out def trace_id(text): h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] PRIORITIES, ACTIONS = ["dusuk", "orta", "yuksek"], ["kayit", "yonlendirme", "bilgi"] SCHEMA = {"talep_no": "sayi", "konu": TOPICS, "oncelik": PRIORITIES, "aksiyon": ACTIONS} PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"), ("faturami odedim ama sistemde borc gorunuyor", "fatura"), ("fatura ayrintisini nereden gorebilirim", "fatura"), ("evde su akmiyor bir ariza var galiba", "ariza"), ("baglanti bozuk oldu ve hic calismiyor", "ariza"), ("ariza kaydi actim ama ekip gelmedi", "ariza"), ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"), ("sayac degisimi icin randevu istiyorum", "sayac"), ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"), ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"), ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"), ("birim ucret abone tipine gore degisiyor mu", "tarife"), ("mahallede kesinti var ne zaman gelecek", "kesinti"), ("planli kesinti duyurusu nereden takip edilir", "kesinti"), ("aksamdan beri kesinti yasiyoruz", "kesinti")] TICKETS = [] for i in range(120): # 120 FICTIONAL support tickets r = rng(SEED + i) topic = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] TICKETS.append({"id": 4001 + i, "topic": topic, "text": [x for x in PATTERNS if x[1] == topic][int(r() * 3)][0]}) CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti"} TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura", "ekip": "kesinti", "takip": "tarife", "giril": "fatura"} # misleading cues COUNTER = [0] def parse_record(text): """Key=value pair parser. Returns None for a response that cannot be parsed.""" if "=" not in text: return None record = {} for part in text.split(";"): if "=" not in part: return None key, _, value = part.partition("=") record[key.strip()] = value.strip() return record def validate(record): """Schema validation: missing key, extra key, invalid value.""" if record is None: return ["ayristirilamadi"] errors = [f"eksik:{k}" for k in SCHEMA if k not in record] errors += [f"fazla:{k}" for k in record if k not in SCHEMA] for key, rule in SCHEMA.items(): if key not in record: continue value = record[key] if rule == "sayi" and not value.isdigit(): errors.append(f"sayi_degil:{key}") elif rule != "sayi" and value not in rule: errors.append(f"gecersiz:{key}") return errors def draw(prompt, seed, salt): """One separately salted draw for each decision; decisions are not tied to each other.""" return rng((seed + salt) ^ trace_id(prompt))() def call(prompt, ticket, mode, seed=SEED): """mode: free | format | schema. In schema mode the generation cannot leave the schema.""" COUNTER[0] += 1 toks = tokenize(prompt) p = {k: 0.15 for k in TOPICS} for t in tokenize(ticket["text"]): # the topic is read only from the ticket text for table, w in ((CUES, 1.0), (TRAPS, 1.3)): if table.get(t) in p: p[table[t]] += w k = max(sorted(p), key=lambda a: p[a]) prio = PRIORITIES[pick(draw(prompt, seed, 11), [0.31, 0.44, 0.25])] act = ACTIONS[pick(draw(prompt, seed, 29), [0.46, 0.32, 0.22])] line = f"talep_no={ticket['id']} ; konu={k} ; oncelik={prio} ; aksiyon={act}" if mode == "schema": # schema imposition return line, k if mode == "free": return f"talebiniz {k} konusunda kaydedildi ve {act} adimi baslatildi", k tendency = 0.62 if "bicim" in toks else 0.0 # format description in the prompt if "ornek" in toks: tendency += 0.22 if draw(prompt, seed, 53) < tendency: return line, k broken = pick(draw(prompt, seed, 71), # response that does not follow the description [0.22, 0.20, 0.18, 0.16, 0.24]) if broken == 0: return f"talebiniz {k} konusunda kaydedildi ve {act} adimi baslatildi", k if broken == 1: return f"talep_no={ticket['id']} ; konu={k} ; aksiyon={act}", k if broken == 2: return f"talep_no={ticket['id']} ; konu={k} ; oncelik=acil ; aksiyon={act}", k if broken == 3: return f"talep_no=TL{ticket['id']} ; konu={k} ; oncelik={prio} ; aksiyon={act}", k return f"talep_no={ticket['id'] + 500} ; konu={k} ; oncelik={prio} ; aksiyon={act}", k FORMAT_DESCRIPTION = ("Yaniti su bicimde yaz: talep_no= ; konu= ; oncelik= ; aksiyon= . ") def build_prompt(prefix, x): return prefix + f"Talep {x['id']} : {x['text']}" print("schema keys:", " , ".join(SCHEMA)) for mode, prefix in (("free", "Su talebi yanitla . "), ("format", FORMAT_DESCRIPTION), ("schema", "Sema: talep_no ; konu ; oncelik ; aksiyon . ")): y, _ = call(build_prompt(prefix, TICKETS[0]), TICKETS[0], mode) errors = validate(parse_record(y)) print(f"mode {mode:<8} {y}") print(f" errors {errors if errors else 'none'}") print("\nfirst four validation errors in format description mode:") n = 0 for x in TICKETS: y, _ = call(build_prompt(FORMAT_DESCRIPTION, x), x, "format") errors = validate(parse_record(y)) if errors and n < 4: n += 1 print(f" {errors[0]:<22} {y[:52]}") ``` ``` schema keys: talep_no , konu , oncelik , aksiyon mode free talebiniz kesinti konusunda kaydedildi ve yonlendirme adimi baslatildi errors ['ayristirilamadi'] mode format talep_no=TL4001 ; konu=kesinti ; oncelik=orta ; aksiyon=kayit errors ['sayi_degil:talep_no'] mode schema talep_no=4001 ; konu=kesinti ; oncelik=orta ; aksiyon=bilgi errors none first four validation errors in format description mode: sayi_degil:talep_no talep_no=TL4001 ; konu=kesinti ; oncelik=orta ; aksi gecersiz:oncelik talep_no=4009 ; konu=sayac ; oncelik=acil ; aksiyon= ayristirilamadi talebiniz kesinti konusunda kaydedildi ve yonlendirm sayi_degil:talep_no talep_no=TL4032 ; konu=tarife ; oncelik=orta ; aksiy ``` The output that the three modes produced on the same ticket shows where the difference lies. The free text has the correct topic in it (`kesinti`, outage) but no record field; the parser finds nothing. In the format description mode the response is almost correct: all four fields are in place, but two letters have come before the ticket number and the field is no longer a number. The record system **rejects** this line, even though the content of the response is entirely correct. The four error examples also say where the breakage is: one falls because a field is not a number, one because it carries a value outside the closed set, one because it could not be parsed at all. All of them are format errors and none is a topic error. ## The Conformance and Cost Table of the Three Levels The same 120 tickets are run with four prompts. The first three ask **by writing into the prompt**, the last one constrains the generation. The second run is taken with seed 20260219 and the schema-conforming rate is printed separately for it. ```python EXAMPLE = "Ornek: talep_no=4000 ; konu=ariza ; oncelik=orta ; aksiyon=kayit . " LEVELS = [("free text", "free", "Su talebi yanitla . "), ("format description", "format", FORMAT_DESCRIPTION), ("format description + example", "format", FORMAT_DESCRIPTION + EXAMPLE), ("imposed schema", "schema", "Sema: talep_no ; konu ; oncelik ; aksiyon . ")] def read_topic(y, record): if record and "konu" in record: return record["konu"] return next((s for s in y.split() if s in TOPICS), None) def measure(mode, prefix, seed=SEED): COUNTER[0] = 0 parsed = valid = hits = recordable = n_in = n_out = 0 tally = {} for x in TICKETS: prompt = build_prompt(prefix, x) y, _ = call(prompt, x, mode, seed) record = parse_record(y) errors = validate(record) parsed += record is not None valid += not errors hit = read_topic(y, record) == x["topic"] hits += hit recordable += hit and not errors for e in errors: tally[e.split(":")[0]] = tally.get(e.split(":")[0], 0) + 1 n_in += len(tokenize(prompt)) n_out += len(tokenize(y)) return (parsed / 120, valid / 120, hits / 120, recordable / 120, n_in, n_out, tally) print("level parseable valid A valid B topic hit recordable input output") for name, mode, prefix in LEVELS: a, b = measure(mode, prefix), measure(mode, prefix, SEED + 1) print(f"{name:<28} {a[0]:>9.4f} {a[1]:>7.4f} {b[1]:>7.4f} {a[2]:>9.4f}" f" {a[3]:>10.4f} {a[4]:>5} {a[5]:>6}") for name, mode, prefix in LEVELS[1:3]: print(f"{name:<28} error distribution {dict(sorted(measure(mode, prefix)[6].items()))}") ``` ``` level parseable valid A valid B topic hit recordable input output free text 0.0000 0.0000 0.0000 0.8833 0.0000 2254 1614 format description 0.9333 0.8000 0.7167 0.8833 0.7000 5614 1789 format description + example 0.9667 0.8833 0.8667 0.8833 0.7750 7654 1789 imposed schema 1.0000 1.0000 1.0000 0.8833 0.8833 2974 1810 format description error distribution {'ayristirilamadi': 8, 'eksik': 1, 'gecersiz': 9, 'sayi_degil': 6} format description + example error distribution {'ayristirilamadi': 4, 'eksik': 5, 'gecersiz': 2, 'sayi_degil': 3} ``` The first row is a complete loss from the record system's point of view. The topic hit rate of the free text is **0.8833**, that is, about eight of every nine responses state the correct topic. The rate of recordable correct records, however, is **0.0000**. The correct information was produced and **ignored because it could not be read**. This is the clearest row showing why format checking is a criterion and not a decoration. The second row is the partial gain of writing into the prompt: parseable responses **0.9333**, but schema-conforming **0.8000**. The 0.1333 in between consists of responses that can be fully parsed but fail validation: in nine the priority is outside the closed set, in six the ticket number is not a number, in one a field is missing. Parseability and validity are not the same thing and cannot be measured with a single number. ## The Unmeasurable Gain and the Reverse Side of the Cost The third row is the one that fits intuition best and sits weakest in the table. Adding an example raises the conformance **from 0.8000 to 0.8833**; the gain is **0.0833**. The two-run columns of the format description row, which is the base of the comparison, need to be looked at: that row gives **0.7167** on the second seed, that is, its two-run difference is also **0.0833**. The gain is **not larger** than the difference. The rule of the course applies directly here: if the difference a prompt change adds does not exceed the two-run difference, it **counts as unmeasured**. Adding an example may have worked, but this table does not show it. The cost of the same change is certain, however. The input tokens go **from 5614 to 7654**: seventeen per ticket, **2040 tokens** in total. A measurable cost paid for an unmeasurable gain is the exact counterpart of the second claim of this course. The fourth row is the opposite of this. The imposed schema raises the conformance to **1.0000**; the gain over the base is **0.2000**, two and a half times the two-run difference. The cost column is even more surprising: the input tokens stay at **2974**. That is a little more than half of the format description and two fifths of the description with the example. This is because the schema is not **explained** to the prompt at length, it is declared briefly with the field names, and the real work is done in constraining the generation. The cheapest prompt gives the highest conformance. The output tokens, on the other hand, are close to each other at the four levels (between 1614 and 1810), because what changes is not the length of the output but its **format**. Finally, the topic hit rate column is fixed at **0.8833** on all four rows, and this is not a coincidence. The imposed schema guarantees the **format** of the response, not its **content**. Fourteen tickets are still assigned to the wrong topic, and this wrong topic now stands inside a perfectly formatted record, as a valid value. Validity is not correctness; a record that passes the schema can be wrong. Indeed, in the format description row, thirteen of the records that pass the schema carry a ticket number that is not on record instead of the number written into the prompt, and the validator sees none of them. ## Summary - The schema is four fields and its validator is written inside the lesson with the standard library; the error classes are counted separately. - In free text the topic hit rate is 0.8833 while the rate of recordable correct records is 0.0000: the correct information is ignored because it cannot be read. - The format description raises the conformance to 0.8000; parseability (0.9333) and conformance to the schema are not the same thing. - Adding an example raises the conformance to 0.8833, but the gain is 0.0833 and the two-run difference of the format description row is also 0.0833: the gain counts as unmeasured and 2040 tokens are paid for it. - The imposed schema raises the conformance to 1.0000 and does so with 2974 input tokens; the cheapest prompt gives the highest conformance. - The topic hit rate is fixed at 0.8833 at all four levels and thirteen records that pass the schema carry a ticket number that is not on record: the schema guarantees the format, not the content. ## Next Step In this lesson the validator was used only as a counter: it counted how many responses fell, and the fallen responses were thrown away. In a production pipeline throwing away is not an option; a fallen record is either recovered or processed by hand. Moreover, the problem seen in the last paragraph was never addressed: a record that passes the schema can violate a **business rule**, and a ticket number that is not on record is the clearest example. The next lesson counts the two checks separately and builds a retry loop on top: how many responses do not conform to the schema, how many get caught on a business rule, how many of these the retry recovers and how many extra calls this recovery costs.