Output Validation

The previous lesson measured the format of the output on three levels and used the validator only as a counter: it counted how many responses fell and threw the fallen ones away. Two gaps were written into its closing. The first: in a production pipeline throwing away a fallen record is not an option. The second: a record that passes the schema can also be wrong. In the format description row of the previous lesson, thirteen of the records that passed the schema carried a ticket number that was not on record instead of the number written into the prompt, and the validator never saw it.

This lesson separates the two checks and builds a retry loop on top. The procedure of input validation, schema and business rule was measured in the Web API Design course and in the Cybersecurity curriculum; it is not repeated here. The only thing added is that the producer is uncertain: a rejected record can be requested again, and the second request can give a different response. This is also what will be measured — what validation catches, what the retry recovers and how many calls it costs.

• OC25. The code is a simulator; no real endpoint is called. The tickets, the fields and the rules are fictional, seed 20260218.
• OC26. The starting prompt is the format description prompt of the previous lesson; the imposed schema is deliberately not used, because this lesson measures what is done with an erroneous output.
• OC27. The two checks are separate. Schema validation looks at the presence of the fields and their value sets. The business rule check is applied only to records that passed the schema and looks at the rules of the institution.
• OC28. Three business rules are fictional: a high-priority record cannot be closed with an information action, an outage record cannot be closed with a record action alone, and the ticket number must be in the set of registered tickets. The schema cannot see any of the three.
• OC29. The retry repeats the same request with a new seed. Because the model is uncertain, the second response can come out different; this is the only basis of the correction.
• OC30. A variant writes the error into the prompt: the error name of the rejected response is added to the prompt. This raises the input tokens.
• OC31. A faulty record that enters the system is counted separately: with no validation the acceptance rate looks like 1.0000, but a part of the records is broken.
• OC32. The cost is written as the number of extra calls and total tokens; for the two-run difference the second run is taken with seed 20260219.

The Error the Schema Cannot See

A schema defines the format of a record: which fields exist and which set the values come from. The rules of an institution, on the other hand, look at the meaning of the record and cannot be written in the language of a schema. This lesson defines three fictional rules and shows that all three are triggered on perfectly formatted records.

# SIMULATOR -- FICTIONAL. No real model endpoint is called. The schema validation and the
# business rule check are written inside the lesson; no library is used from outside.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(text):                     # FICTIONAL tokenizer, a piece has at most five characters
    out = []
    for piece in text.lower().replace(",", " , ").split():
        while len(piece) > 5:
            out.append(piece[:5])
            piece = piece[5:]
        out.append(piece)
    return out


def trace_id(text):
    h = 2166136261
    for c in text:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
PRIORITIES, ACTIONS = ["dusuk", "orta", "yuksek"], ["kayit", "yonlendirme", "bilgi"]
SCHEMA = {"talep_no": "sayi", "konu": TOPICS, "oncelik": PRIORITIES, "aksiyon": ACTIONS}
PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"),
            ("faturami odedim ama sistemde borc gorunuyor", "fatura"),
            ("fatura ayrintisini nereden gorebilirim", "fatura"),
            ("evde su akmiyor bir ariza var galiba", "ariza"),
            ("baglanti bozuk oldu ve hic calismiyor", "ariza"),
            ("ariza kaydi actim ama ekip gelmedi", "ariza"),
            ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"),
            ("sayac degisimi icin randevu istiyorum", "sayac"),
            ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"),
            ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"),
            ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"),
            ("birim ucret abone tipine gore degisiyor mu", "tarife"),
            ("mahallede kesinti var ne zaman gelecek", "kesinti"),
            ("planli kesinti duyurusu nereden takip edilir", "kesinti"),
            ("aksamdan beri kesinti yasiyoruz", "kesinti")]
TICKETS = []
for i in range(120):                    # 120 FICTIONAL support tickets
    r = rng(SEED + i)
    topic = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    TICKETS.append({"id": 4001 + i, "topic": topic,
                    "text": [x for x in PATTERNS if x[1] == topic][int(r() * 3)][0]})

CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza",
        "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac",
        "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife",
        "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti",
        "mahal": "kesinti"}
TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura",
         "ekip": "kesinti", "takip": "tarife", "giril": "fatura"}   # misleading cues
COUNTER = [0]


def parse_record(text):
    """Key=value pair parser. Returns None for a response that cannot be parsed."""
    if "=" not in text:
        return None
    record = {}
    for part in text.split(";"):
        if "=" not in part:
            return None
        key, _, value = part.partition("=")
        record[key.strip()] = value.strip()
    return record


def validate(record):
    """Schema validation: missing key, extra key, invalid value."""
    if record is None:
        return ["ayristirilamadi"]
    errors = [f"eksik:{k}" for k in SCHEMA if k not in record]
    errors += [f"fazla:{k}" for k in record if k not in SCHEMA]
    for key, rule in SCHEMA.items():
        if key not in record:
            continue
        value = record[key]
        if rule == "sayi" and not value.isdigit():
            errors.append(f"sayi_degil:{key}")
        elif rule != "sayi" and value not in rule:
            errors.append(f"gecersiz:{key}")
    return errors


def draw(prompt, seed, salt):
    """One separately salted draw for each decision; decisions are not tied to each other."""
    return rng((seed + salt) ^ trace_id(prompt))()


def call(prompt, ticket, seed=SEED):
    """The format description prompt of the previous lesson; a share of the responses do not follow it."""
    COUNTER[0] += 1
    toks = tokenize(prompt)
    p = {k: 0.15 for k in TOPICS}
    for t in tokenize(ticket["text"]):      # the topic is read only from the ticket text
        for table, w in ((CUES, 1.0), (TRAPS, 1.3)):
            if table.get(t) in p:
                p[table[t]] += w
    k = max(sorted(p), key=lambda a: p[a])
    prio = PRIORITIES[pick(draw(prompt, seed, 11), [0.31, 0.44, 0.25])]
    act = ACTIONS[pick(draw(prompt, seed, 29), [0.46, 0.32, 0.22])]
    line = f"talep_no={ticket['id']} ; konu={k} ; oncelik={prio} ; aksiyon={act}"
    tendency = 0.62 if "bicim" in toks else 0.0         # format description in the prompt
    if "hatal" in toks:                                 # retry that writes the error into the prompt
        tendency += 0.14
    if draw(prompt, seed, 53) < tendency:
        return line
    broken = pick(draw(prompt, seed, 71),               # response that does not follow the description
                  [0.22, 0.20, 0.18, 0.16, 0.24])
    if broken == 0:
        return f"talebiniz {k} konusunda kaydedildi ve {act} adimi baslatildi"
    if broken == 1:
        return f"talep_no={ticket['id']} ; konu={k} ; aksiyon={act}"
    if broken == 2:
        return f"talep_no={ticket['id']} ; konu={k} ; oncelik=acil ; aksiyon={act}"
    if broken == 3:
        return f"talep_no=TL{ticket['id']} ; konu={k} ; oncelik={prio} ; aksiyon={act}"
    return f"talep_no={ticket['id'] + 500} ; konu={k} ; oncelik={prio} ; aksiyon={act}"


FORMAT_DESCRIPTION = ("Yaniti su bicimde yaz: talep_no=<sayi> ; konu=<fatura|ariza|sayac|"
                      "tarife|kesinti> ; oncelik=<dusuk|orta|yuksek> ; aksiyon=<kayit|"
                      "yonlendirme|bilgi> . ")


def build_prompt(prefix, x):
    return prefix + f"Talep {x['id']} : {x['text']}"


REGISTERED = {t["id"] for t in TICKETS}


def business_rules(record):
    """Institutional rules of a record that passed the schema. The schema sees none of them."""
    errors = []
    if record.get("oncelik") == "yuksek" and record.get("aksiyon") == "bilgi":
        errors.append("yuksek_oncelik_bilgi_ile_kapanamaz")
    if record.get("konu") == "kesinti" and record.get("aksiyon") == "kayit":
        errors.append("kesinti_kayitla_kapanamaz")
    if record.get("talep_no", "").isdigit() and int(record["talep_no"]) not in REGISTERED:
        errors.append("kayitli_olmayan_talep_no")
    return errors


schema_fail, rule_fail, seen = 0, 0, {}
for x in TICKETS:
    y = call(build_prompt(FORMAT_DESCRIPTION, x), x)
    record = parse_record(y)
    errors = validate(record)
    if errors:
        schema_fail += 1
        continue
    rules = business_rules(record)
    rule_fail += bool(rules)
    if rules:
        seen.setdefault(rules[0], y)
print(f"FICTIONAL tickets {len(TICKETS)}: caught by schema {schema_fail}, "
      f"caught by business rule {rule_fail}")
for e, y in sorted(seen.items()):
    print(f"  {e}\n    {y}")

FICTIONAL tickets 120: caught by schema 24, caught by business rule 24
  kayitli_olmayan_talep_no
    talep_no=4505 ; konu=sayac ; oncelik=orta ; aksiyon=kayit
  kesinti_kayitla_kapanamaz
    talep_no=4015 ; konu=kesinti ; oncelik=dusuk ; aksiyon=kayit
  yuksek_oncelik_bilgi_ile_kapanamaz
    talep_no=4022 ; konu=tarife ; oncelik=yuksek ; aksiyon=bilgi

The two numbers are equal to each other, and this equality is the starting point of the lesson: 24 responses do not conform to the schema, and 24 of the 96 responses that pass the schema are caught on a business rule. That is, the format check sees only half of the errors. All three examples are perfectly formatted; the parser and the validator find nothing in any of them. In the first the ticket number is a number not written in the prompt, in the second an outage record has been closed with the wrong action, in the third a high-priority record has been tied to an information action.

These three rules cannot be translated into the language of the schema, because none of them concerns the value of a single field. The first two are a consistency between fields, the third is the consistency of the record with the outside world. Without counting these two classes separately in the output of an uncertain producer, validation does not count as measured.

What the Retry Recovers

Six pipelines are run on the same 120 tickets. The first three have no retry and only measure what the check eliminates; the last three request the rejected record again.

PIPELINES = [("no validation", False, False, 0, False),
             ("schema only", True, False, 0, False),
             ("schema + business rule", True, True, 0, False),
             ("+ 1 retry", True, True, 1, False),
             ("+ 2 retries", True, True, 2, False),
             ("+ 1 retry, error in prompt", True, True, 1, True)]


def check(record, with_schema, with_rules):
    errors = validate(record) if with_schema else []
    if not errors and with_rules and record is not None:
        errors = business_rules(record)
    return errors


def run_pipeline(with_schema, with_rules, retries, write_error, seed=SEED):
    COUNTER[0] = 0
    first_schema = first_rule = recovered = accepted = faulty = extra = tok = 0
    for x in TICKETS:
        prompt = build_prompt(FORMAT_DESCRIPTION, x)
        y = call(prompt, x, seed)
        tok += len(tokenize(prompt)) + len(tokenize(y))
        record = parse_record(y)
        first_schema += bool(validate(record))
        first_rule += not validate(record) and bool(business_rules(record))
        errors = check(record, with_schema, with_rules)
        for attempt in range(retries):
            if not errors:
                break
            extra += 1
            prompt2 = prompt + f" Onceki yanit hatali: {errors[0]} ." if write_error else prompt
            y = call(prompt2, x, seed + 1 + attempt)
            tok += len(tokenize(prompt2)) + len(tokenize(y))
            record = parse_record(y)
            errors = check(record, with_schema, with_rules)
            recovered += not errors
        accepted += not errors
        faulty += (not errors) and bool(validate(record) or business_rules(record))
    return (first_schema, first_rule, recovered, accepted / 120, faulty / 120, extra, tok)


print("pipeline                   schema  rule  recovered  accepted  faulty  extra calls  tokens")
for name, s, r, n, w in PIPELINES:
    a = run_pipeline(s, r, n, w)
    print(f"{name:<26} {a[0]:>6}  {a[1]:>4}  {a[2]:>9}  {a[3]:>8.4f}  {a[4]:>6.4f}"
          f"  {a[5]:>11}  {a[6]:>6}")
b = run_pipeline(True, True, 1, False, SEED + 1)
print(f"second run (+ 1 retry): accepted {b[3]:.4f}, recovered {b[2]}, "
      f"extra calls {b[5]}")

pipeline                   schema  rule  recovered  accepted  faulty  extra calls  tokens
no validation                  24    24          0    1.0000  0.4000            0    7403
schema only                    24    24          0    0.8000  0.2000            0    7403
schema + business rule         24    24          0    0.6000  0.0000            0    7403
+ 1 retry                      24    24         26    0.8167  0.0000           48   10350
+ 2 retries                    24    24         34    0.8833  0.0000           70   11716
+ 1 retry, error in prompt     24    24         23    0.7917  0.0000           48   10864
second run (+ 1 retry): accepted 0.7500, recovered 26, extra calls 56

The first three rows were produced with a single call each, and the token column is the same in all three: 7403. The only thing that changes is the check made before acceptance. With no validation the acceptance rate looks like 1.0000 and the rate of wrong records entering the record system is 0.4000. That is, when nothing is measured the system looks flawless, and two of every five records are broken. When the schema is added the faulty records fall to 0.2000, and when the business rule is added as well, to 0.0000.

The price of this is the acceptance rate: from 1.0000 to 0.6000. Validation corrects no record; it only makes them visible. The system in the third row is better than the first but does less work; 48 tickets are left on hand. The measure of validation is not the acceptance rate but how much of what is accepted is correct.

The Cost of Recovery

The fourth row adds the retry. 26 records are recovered and the acceptance rate rises from 0.6000 to 0.8167; the faulty record rate stays at 0.0000, because the recovery has been placed behind the check, not in front of it. The cost is 48 extra calls and 2947 tokens: 1.85 calls and 113.3 tokens per recovered record. Since the gain is 0.2167 and the two-run difference is 0.0667, this row is comfortably measurable.

The fifth row is where the rule of the course is applied most sharply. The second retry recovers eight more records and raises the acceptance rate to 0.8833. The gain is 0.0666. The two-run difference of the previous pipeline is 0.0667: 0.8167 in the first run, 0.7500 in the second. The gain is not larger than the difference, so it counts as unmeasured. The cost paid for this row, on the other hand, is certain: 22 extra calls and 1366 tokens, 2.75 calls per recovered record. An unmeasurable gain has been bought with a measurable cost.

The sixth row tests the correction that best fits intuition: writing the error of the rejected response into the prompt. The result is 23 recoveries instead of 26 and 0.7917 in the acceptance rate. Since the 0.0250 in between stays far below the two-run difference of 0.0667, there is no measurable difference between this row and the fourth. The cost is measurable, however: with the same 48 calls the tokens rise from 10350 to 10864, that is, 514 tokens have been added. Writing the errors into the prompt is here not an improvement but an unmeasured guess.

It is wrapped up with three numbers. The previous prompt is the format description prompt: acceptance 0.6000, faulty record 0.0000, 7403 tokens, 120 calls. The changed pipeline adds one retry: acceptance 0.8167, the same faulty record rate, 10350 tokens, 168 calls. The cost is 48 calls and 2947 tokens; for the first time in this course a scaffolding moves the number of calls, and the larger part of the cost sits there.

Summary

• Schema validation eliminates 24 responses, the business rule check eliminates 24 more of the 96 responses that pass the schema: the format check sees half of the errors.
• All three business rules are triggered on perfectly formatted records; two are consistency between fields, one is the consistency of the record with the outside world.
• With no validation the acceptance rate is 1.0000 and the faulty record entering the system is 0.4000; the two checks bring the faulty record down to 0.0000 while lowering the acceptance to 0.6000. Validation does not correct, it makes visible.
• One retry recovers 26 records and raises the acceptance to 0.8167; its cost is 48 extra calls and 2947 tokens, 1.85 calls per recovered record.
• The gain of the second retry is 0.0666 and the two-run difference of the pipeline is 0.0667: the gain counts as unmeasured and 22 calls and 1366 tokens are paid for it.
• Writing the error into the prompt recovers 23 records instead of 26 and adds 514 tokens; there is no measurable difference between the two pipelines.

Next Step

In this lesson the recovery was done with a single lever: asking the same request again. The 14 records that could not be recovered were never examined. Some of them will not be corrected however much is tried, because the problem is not in the draw of the response but in something the model does not know. If the tariff tier a ticket falls into is not written into the prompt, a better instruction cannot produce it. The next lesson counts this distinction: the same wrong response is sometimes corrected by a better instruction and sometimes only by supplying the missing information. Which error is corrected by which is counted separately and the cost of the two is compared.