Clear and Measurable Requests

The previous lesson added an identity and a procedure to the prompt and found that the difference of none of the four prefixes is measurable. Throughout that measurement one thing never changed: what was asked. The request was the same sentence in every prompt, and that sentence itself was never questioned. Yet the task definition is not the most expensive part of the prompt but its most decisive part.

The cost of an ambiguous request is not that it gives an ambiguous response. An ambiguous request does not produce a single wrong answer; it produces a distribution. The same prompt, the same record, the same model, and in ten separate runs a few separate responses that do not replace one another. This distribution is not visible in any measure that looks at a single run; what shows in a single run is one sample drawn from that distribution. This lesson measures the distribution and counts how far making the request measurable narrows it.

• PD26. The code calls a simulator; no real model endpoint is called. The support tickets are fictional and are the first 36 records of the dataset in the previous lessons. The dataset was made smaller because every record is called ten times.
• PD27. Every record is measured with ten runs: seeds 20260218 to 20260227. The distribution measures are the total of these ten responses, not of a single run.
• PD28. The number of distinct responses is the number of response texts that come out different from one another in ten runs, averaged per record. Its smallest value is 1 and its largest is 10.
• PD29. The top share is the share of the most repeated response in ten runs. It approaches 1 in a narrow distribution and 0.1 in a wide one.
• PD30. The rate limit fires on every thirty-seventh call and returns an empty response. So that it does not disturb the distribution count, the call counter is reset before the ten runs of every record; this is a measurement decision and it is in the code.
• PD31. The four request forms are built incrementally: a vague verb, a clear verb, a clear verb with a closed option set, and that with an ambiguity rule added.
• PD32. In this lesson the form of the response is not requested in any prompt. What is measured is the clarity of the request; the form of the output is the subject of the next lesson.
• PD33. The hit rate is computed separately in each of the ten runs; the table prints its mean and its band (the difference between the largest and the smallest run).
• PD34. The cost is in tokens and is written per request as input and output; the cost of ten runs is ten times that.

The Distribution an Ambiguous Request Produces

The first block sets up the simulator and the fiction, defines the four request forms, calls a single fictional record ten times with the most ambiguous form and counts the responses that come out.

# SIMULATOR. This code does not call a real model endpoint; it is a language model
# simulator written with the standard library. The support tickets are FICTIONAL too.
import math

SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW, MAX_OUT = 240, 24           # context window and maximum output, in tokens


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):                    # FICTIONAL tokenizer: pieces of at most five characters
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def prompt_trace(m):                # trace that depends on the prompt content, independent of the run
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
TEMPLATES = {
    "fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi",
               "faturami odedim ama sistemde borc gorunuyor",
               "fatura ayrintisini nereden gorebilirim"],
    "ariza": ["evde su akmiyor bir ariza var galiba",
              "baglanti bozuk oldu ve hic calismiyor",
              "ariza kaydi actim ama ekip gelmedi"],
    "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor",
              "sayac degisimi icin randevu istiyorum",
              "sayac uzerindeki endeks ile fatura ayni degil"],
    "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum",
               "tarife degisikligi icin ne yapmam gerekiyor",
               "birim ucret abone tipine gore degisiyor mu"],
    "kesinti": ["mahallede kesinti var ne zaman gelecek",
                "planli kesinti duyurusu nereden takip edilir",
                "aksamdan beri kesinti yasiyoruz"],
}
EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim",
          "acil bilgi almam gerekiyor", "kayit numaram elimde yok"]

TICKETS = []
for i in range(120):
    r = rng(SEED + i)
    k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    m = TEMPLATES[k][int(r() * 3)]
    if r() < 0.34:
        m += " " + EXTRAS[int(r() * 4)]
    TICKETS.append({"id": 4001 + i, "text": m, "topic": k})

CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"]
CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura",
        "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza",
        "sayac": "sayac", "endek": "sayac", "okuma": "sayac",
        "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife",
        "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti",
        "borc": "odeme", "odedi": "odeme", "ucret": "odeme",
        "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet",
        "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"}
TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura",
         "ekip": "kesinti", "takip": "tarife", "giril": "fatura"}   # misleading cues
FREE_FORM = ["talebiniz {k} konusuyla ilgili gorunuyor",
             "bu kayit {k} basligina girer , ilgili ekibe yonlendirildi",
             "kaydiniz alindi , konuyu inceleyip donus yapacagiz"]
CALL_COUNT = [0]


def call(prompt, seed=SEED, temperature=0.6, max_out=MAX_OUT):
    CALL_COUNT[0] += 1
    tok = tokenize(prompt)
    if len(tok) > WINDOW:
        return {"status": "context_overflow", "tokens_in": len(tok), "tokens_out": 0, "response": ""}
    if CALL_COUNT[0] % 37 == 0:
        return {"status": "rate_limit", "tokens_in": len(tok), "tokens_out": 0, "response": ""}
    r = rng(seed ^ prompt_trace(prompt))
    if len(tok) > 90 and r() < 0.04:
        return {"status": "timeout", "tokens_in": len(tok), "tokens_out": 0, "response": ""}
    cand = [k for k in CONCEPTS if k[:5] in tok]    # concepts named in the prompt
    if len(cand) < 3:
        cand = CONCEPTS
    p = {k: 0.15 for k in cand}
    for t in tok:
        for table, w in ((CUES, 1.0), (TRAPS, 1.3)):
            if table.get(t) in p:
                p[table[t]] += w
    if temperature <= 0.0:
        k = max(sorted(p), key=lambda a: p[a])       # most likely choice
    else:
        e = [math.exp(p[a] / temperature) for a in sorted(p)]
        k = sorted(p)[pick(r(), [v / sum(e) for v in e])]
    y = k if "tek" in tok else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k)
    ot = tokenize(y)
    if len(ot) > max_out:                            # stop condition
        ot = ot[:max_out]
        y = " ".join(ot)
    return {"status": "ok", "tokens_in": len(tok), "tokens_out": len(ot), "response": y}


def read_label(y):                                   # extract a label from the response
    for s in y.lower().split():
        if s in TOPICS:
            return s
    return None


VERB = "Su destek talebini konusuna gore etiketle ."
SET_TEXT = " Etiketler: fatura , ariza , sayac , tarife , kesinti ."
REQUEST_FORMS = (("vague verb", "Su destek talebini degerlendir ."),
                 ("clear verb", VERB),
                 ("closed set", VERB + SET_TEXT),
                 ("ambiguity rule",
                  VERB + SET_TEXT + " Birden cok konu varsa ana konuyu sec ."))
DATASET = TICKETS[:36]
SEEDS = [SEED + i for i in range(10)]


def runs(head, x):
    CALL_COUNT[0] = 0       # reset for every ticket so the rate limit does not disturb the distribution count
    return [call(head + " Talep: " + x["text"], seed=t) for t in SEEDS]


def breakdown(name, head, x):
    s = [c["response"] for c in runs(head, x)]
    print(f"{name}, ten runs -> {len(set(s))} distinct responses")
    for v in sorted(set(s)):
        print(f"  {s.count(v)}/10  {v}")


print(f"{'request':<19} {'tokens':>9}")
for name, h in REQUEST_FORMS:
    print(f"{name:<19} {len(tokenize(h)):>9d}")
x0 = DATASET[3]
print(f"FICTIONAL ticket {x0['id']} (true topic {x0['topic']}): {x0['text']}")
breakdown(REQUEST_FORMS[0][0], REQUEST_FORMS[0][1], x0)

request                tokens
vague verb                  9
clear verb                 11
closed set                 26
ambiguity rule             36
FICTIONAL ticket 4004 (true topic ariza): ariza kaydi actim ama ekip gelmedi
vague verb, ten runs -> 7 distinct responses
  1/10  bu kayit ariza basligina girer , ilgili ekibe yonlendirildi
  1/10  bu kayit odeme basligina girer , ilgili ekibe yonlendirildi
  1/10  bu kayit sayac basligina girer , ilgili ekibe yonlendirildi
  2/10  kaydiniz alindi , konuyu inceleyip donus yapacagiz
  1/10  talebiniz fatura konusuyla ilgili gorunuyor
  2/10  talebiniz kesinti konusuyla ilgili gorunuyor
  2/10  talebiniz sikayet konusuyla ilgili gorunuyor

A single record gave seven distinct responses in ten runs. The most frequent response was repeated in only two runs. One of the responses carries no label at all; two others use the labels odeme (payment) and sikayet (complaint), that is, names that have no counterpart in the fictional document vocabulary of the field team. The prompt said “evaluate”, and that verb never said which set a result was wanted from or in what form. The model filled this gap in a different way across the runs. The real meaning of this table lies not in one record but in the distribution across records. An observer who looks at only one of the seven distinct responses reaches the conclusion that the prompt worked or did not work with equal ease.

Four Requests Side by Side

The second block measures the four request forms on thirty-six records and with ten runs. In every row stand the mean number of distinct responses per record, the share of the most frequent response, the unlabeled response share, the mean hit rate of the ten runs, the band of the hit rate and the token cost.

def measure(head):
    dr = tp = unl = ti = to = 0
    hits = [0] * len(SEEDS)
    for x in DATASET:
        c = runs(head, x)
        s = [v["response"] for v in c]
        e = [read_label(v) for v in s]
        dr += len(set(s))
        tp += max(s.count(v) for v in s) / len(s)
        unl += sum(v is None for v in e) / len(e)
        for j, v in enumerate(e):
            hits[j] += v == x["topic"]
        ti += sum(v["tokens_in"] for v in c) / len(c)
        to += sum(v["tokens_out"] for v in c) / len(c)
    n = len(DATASET)
    hr = [v / n for v in hits]
    return (dr / n, tp / n, unl / n, sum(hr) / len(hr), max(hr) - min(hr),
            ti / n, to / n)


print(f"{'request':<19} {'tok':>4} {'distinct':>11} {'top share':>9} {'unlabeled':>10} "
      f"{'hit rate':>8} {'band':>7} {'in':>6} {'out':>6}")
for name, h in REQUEST_FORMS:
    dr, tp, unl, hr, band, ti, to = measure(h)
    print(f"{name:<19} {len(tokenize(h)):>4d} {dr:>11.2f} {tp:>9.4f} {unl:>10.4f} "
          f"{hr:>8.4f} {band:>7.4f} {ti:>6.1f} {to:>6.1f}")
print()
breakdown(REQUEST_FORMS[2][0], REQUEST_FORMS[2][1], x0)

request              tok    distinct top share  unlabeled hit rate    band     in    out
vague verb             9        4.81    0.3944     0.4333   0.4611  0.3333   23.1   11.4
clear verb            11        4.53    0.4333     0.4500   0.4417  0.3333   25.1   11.3
closed set            26        3.89    0.4083     0.2889   0.5306  0.2778   40.1   11.5
ambiguity rule        36        3.78    0.4611     0.3722   0.4722  0.3333   50.1   11.6

closed set, ten runs -> 4 distinct responses
  2/10  bu kayit kesinti basligina girer , ilgili ekibe yonlendirildi
  3/10  kaydiniz alindi , konuyu inceleyip donus yapacagiz
  2/10  talebiniz ariza konusuyla ilgili gorunuyor
  3/10  talebiniz kesinti konusuyla ilgili gorunuyor

Clarifying the Verb Is Not Measurability

The second row is the result of this lesson that is the easiest to misunderstand. The verb “evaluate” was replaced with “label by its topic”; the request is clear as grammar and explicit as an action. The number of distinct responses fell from 4.81 to 4.53, the hit rate fell from 0.4611 to 0.4417, and the unlabeled response share rose from 0.4333 to 0.4500. The two-token addition did not narrow the distribution in any measurable way.

The reason is structural. A clear verb says what to do; it does not say which set to choose from. A prompt that says “label” leaves it to the model where the label will come from, and in every run the model produces a name from its own vocabulary. For a request to be measurable, the verb does not need to be sharp; the set from which the answer can be taken needs to be closed.

The third row shows this. When the names of the five labels are written into the prompt, the number of distinct responses falls to 3.89, the unlabeled share drops from 0.4333 to 0.2889 and the hit rate rises to 0.5306. The cost is plainly countable: the input per request rises from 25.1 to 40.1 tokens, that is, 15 tokens. Over thirty-six records that is 540 and over ten runs 5400 tokens. The ten runs of the same record with the closed set are at the end of the block output: seven distinct responses have fallen to four and the names outside the set have disappeared. On this record the concentration happened on the wrong label; narrowing of the distribution is not a guarantee of hit rate, and the two numbers must stand in separate columns. The fourth row pays off the second claim of the course. The sentence “if many topics, pick the main one” adds ten tokens and lowers the number of distinct responses from 3.89 to 3.78. The same sentence raises the unlabeled share from 0.2889 to 0.3722 and drops the hit rate from 0.5306 to 0.4722. A sentence written to reduce ambiguity may, when measured, have increased it.

The Distribution Is Not Visible in a Single Run

The band column of the table is the methodological result of this lesson. The hit rate bands of the four requests across ten runs are 0.3333, 0.3333, 0.2778 and 0.3333 respectively. The effect of the closed set on the hit rate is +0.0695; exactly a quarter of its own band. Read by the criterion of the previous lesson, this difference counts as unmeasured. That is, which of the four requests is better cannot be told apart in any way with the hit rate read in a single run. The numbers that can tell them apart are different, and all of them are totals over ten runs: the number of distinct responses falls from 4.81 to 3.89 and the unlabeled share from 0.4333 to 0.2889. These two measures cannot be computed from a single run, because in a single run every record has one response and it is not visible which of how many possibilities that response is. The cost of an ambiguous request is a cost that cannot be measured in a single run; this is why most prompt trials never see it and close with a single successful sample.

Summary

• An ambiguous request produces not a single wrong response but a distribution. A single record gave seven distinct responses in ten runs; across the dataset the result was a mean of 4.81 distinct responses per record and a top share of only 0.3944.
• Clarifying the verb is not measurability: the two-token addition lowered the distinct responses to 4.53, dropped the hit rate to 0.4417 and raised the unlabeled share to 0.4500.
• The decision that makes the request measurable is closing the option set: 3.89 distinct responses, an unlabeled share of 0.2889, a hit rate of 0.5306. Its cost is 15 tokens per request, 5400 tokens over ten runs.
• Adding an ambiguity rule added ten tokens and took the unlabeled share to 0.3722 and the hit rate to 0.4722: the sentence written to reduce ambiguity increased it when measured.
• The band of the hit rate across ten runs is between 0.2778 and 0.3333; the +0.0695 effect of the closed set is inside the band. The distribution measures cannot be computed from a single run.

Next Step

In four lessons the components of the prompt were separated, the repetition cost of the persistent instruction was counted, the difference of the role prefixes was found unmeasurable and the request was tied to a closed set. Even so, the prompt with the closed set still produces a mean of 3.89 distinct responses per record, and no label can be read from 0.2889 of the responses. The source of the remaining distribution is no longer what is asked but how it is said: the same label is given in three separate sentence patterns, and sometimes with no label name at all. The request was clarified, but the form of the response is still left to the model. The next lesson takes this up: when the output form is requested, how many responses can be parsed by a machine, and how much of the hit rate the length limit takes back.