Specifying Role and Behavior

The previous lesson measured the length of the instruction and found that the hit rate saturates at thirty-three tokens; the twenty-nine tokens added after that raised the cost for certain and did not move the number in any measurable way. The question of what to add to the instruction was not closed there. The most frequent answer is to tell the model who it is: an area of expertise, a job title, a level of seniority. That this addition raises the hit rate is a common belief.

This lesson measures that belief and, while measuring, puts the third rule of the course to work by name. The model is an uncertain component: the same prompt gives a different number when the seed changes. If the difference that a prompt change adds is not larger than the range the same prompt moves from run to run, that difference is not a gain but a coincidence. In this course such a number counts as unmeasured. The lesson builds this criterion, applies it to four prefixes and writes the result openly.

• PD17. The code calls a simulator; no real model endpoint is called. The support tickets are fictional and are the same 120 records as in the previous two lessons.
• PD18. The base prompt is the medium instruction measured in the previous lesson: the label set, the task and the response form. In this lesson the instruction does not change; only a prefix is added in front of it.
• PD19. Role prompting (specifying a role) is text that attributes an identity to the model. Specifying a behavior is text that writes the steps the model is asked to follow. Both are placed in front of the instruction in the same way.
• PD20. Every prompt is run with four seeds (20260218 to 20260221). The first two seeds are the two runs columns; the difference between the largest and the smallest of the four is the run band.
• PD21. The effect of a prefix is the difference between the mean of its first two runs and the mean of the first two runs of the base.
• PD22. Measurability criterion: an effect that is not larger than the larger of the bands of the two compared prompts counts as unmeasured. The criterion is conservative in this respect and its reason is written in the lesson.
• PD23. The cost is in tokens. The prefix cost is prefix length times number of requests and it is included in the total tokens; the cost per correct label is printed separately.
• PD24. The trace table of the simulator is in the code and is readable. Which label a prefix text adds weight to is verified from this table.
• PD25. This lesson measures only the hit rate and the cost; the form of the response was closed in the previous lesson and is fixed here.

In a Single Ticket the Role Seems to Have Worked

The first block sets up the simulator and the fiction, defines the four prefixes, writes the cost of each over one hundred twenty requests and calls a single fictional ticket with the base and with the role prefix, with two separate seeds.

# SIMULATOR. This code does not call a real model endpoint; it is a language model
# simulator written with the standard library. The support tickets are FICTIONAL too.
import math

SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW, MAX_OUT = 240, 24           # context window and maximum output, in tokens


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):                    # FICTIONAL tokenizer: pieces of at most five characters
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def prompt_trace(m):                # trace that depends on the prompt content, independent of the run
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
TEMPLATES = {
    "fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi",
               "faturami odedim ama sistemde borc gorunuyor",
               "fatura ayrintisini nereden gorebilirim"],
    "ariza": ["evde su akmiyor bir ariza var galiba",
              "baglanti bozuk oldu ve hic calismiyor",
              "ariza kaydi actim ama ekip gelmedi"],
    "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor",
              "sayac degisimi icin randevu istiyorum",
              "sayac uzerindeki endeks ile fatura ayni degil"],
    "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum",
               "tarife degisikligi icin ne yapmam gerekiyor",
               "birim ucret abone tipine gore degisiyor mu"],
    "kesinti": ["mahallede kesinti var ne zaman gelecek",
                "planli kesinti duyurusu nereden takip edilir",
                "aksamdan beri kesinti yasiyoruz"],
}
EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim",
          "acil bilgi almam gerekiyor", "kayit numaram elimde yok"]

TICKETS = []
for i in range(120):
    r = rng(SEED + i)
    k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    m = TEMPLATES[k][int(r() * 3)]
    if r() < 0.34:
        m += " " + EXTRAS[int(r() * 4)]
    TICKETS.append({"id": 4001 + i, "text": m, "topic": k})

CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"]
CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura",
        "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza",
        "sayac": "sayac", "endek": "sayac", "okuma": "sayac",
        "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife",
        "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti",
        "borc": "odeme", "odedi": "odeme", "ucret": "odeme",
        "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet",
        "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"}
TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura",
         "ekip": "kesinti", "takip": "tarife", "giril": "fatura"}   # misleading cues
FREE_FORM = ["talebiniz {k} konusuyla ilgili gorunuyor",
             "bu kayit {k} basligina girer , ilgili ekibe yonlendirildi",
             "kaydiniz alindi , konuyu inceleyip donus yapacagiz"]
CALL_COUNT = [0]


def call(prompt, seed=SEED, temperature=0.6, max_out=MAX_OUT):
    CALL_COUNT[0] += 1
    tok = tokenize(prompt)
    if len(tok) > WINDOW:
        return {"status": "context_overflow", "tokens_in": len(tok), "tokens_out": 0, "response": ""}
    if CALL_COUNT[0] % 37 == 0:
        return {"status": "rate_limit", "tokens_in": len(tok), "tokens_out": 0, "response": ""}
    r = rng(seed ^ prompt_trace(prompt))
    if len(tok) > 90 and r() < 0.04:
        return {"status": "timeout", "tokens_in": len(tok), "tokens_out": 0, "response": ""}
    cand = [k for k in CONCEPTS if k[:5] in tok]    # concepts named in the prompt
    if len(cand) < 3:
        cand = CONCEPTS
    p = {k: 0.15 for k in cand}
    for t in tok:
        for table, w in ((CUES, 1.0), (TRAPS, 1.3)):
            if table.get(t) in p:
                p[table[t]] += w
    if temperature <= 0.0:
        k = max(sorted(p), key=lambda a: p[a])       # most likely choice
    else:
        e = [math.exp(p[a] / temperature) for a in sorted(p)]
        k = sorted(p)[pick(r(), [v / sum(e) for v in e])]
    y = k if "tek" in tok else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k)
    ot = tokenize(y)
    if len(ot) > max_out:                            # stop condition
        ot = ot[:max_out]
        y = " ".join(ot)
    return {"status": "ok", "tokens_in": len(tok), "tokens_out": len(ot), "response": y}


def read_label(y):                                   # extract a label from the response
    for s in y.lower().split():
        if s in TOPICS:
            return s
    return None


INSTRUCTION = ("Etiketler: fatura , ariza , sayac , tarife , kesinti ."
               " Talebi bu etiketlerden birine ata . Yanit olarak tek etiket yaz .")
PREFIXES = (("base", ""),
            ("role A", "Sen bir abone destek gorevlisisin ."),
            ("role B", "Sen bir deneyimli saha teknisyenisin ."),
            ("role C", "Sen dagitim sirketinde on yildir calisan kidemli bir kayit uzmanisin ."),
            ("behavior", "Once talebin ana nesnesini bul , sonra etiketi sec ."
                         " Emin olmadigin durumda talepte gecen ana sozcugu esas al ."))
SEEDS = (SEED, SEED + 1, SEED + 2, SEED + 3)


def build_prompt(p, x):
    return (p + " " if p else "") + INSTRUCTION + " Talep: " + x["text"]


print(f"{'prefix':<9} {'tokens':>9} {'in 120 requests':>15}")
for name, p in PREFIXES:
    b = len(tokenize(p))
    print(f"{name:<9} {b:>9d} {b * len(TICKETS):>15d}")
x0 = TICKETS[2]
print(f"FICTIONAL ticket {x0['id']} ({x0['topic']}): {x0['text']}")
for name, p in PREFIXES[:2]:
    CALL_COUNT[0] = 0
    y = [call(build_prompt(p, x0), seed=t)["response"] for t in SEEDS[:2]]
    print(f"{name:<9} two runs: {y[0]} / {y[1]}")

prefix       tokens in 120 requests
base              0               0
role A            9            1080
role B            9            1080
role C           17            2040
behavior         27            3240
FICTIONAL ticket 4003 (ariza): evde su akmiyor bir ariza var galiba ayrica gecen ay da ayni sorun olmustu
base      two runs: kesinti / ariza
role A    two runs: ariza / ariza

The last two lines show how a prompt change comes to look proven. The real topic of the ticket is fault. The base prompt answered wrong once and right once in two runs; when the role prefix was added, the answer came right in both runs. Anyone looking at the single ticket would conclude “the role raised the hit rate”. The only ground for this conclusion is that the base already moves from run to run. The same demonstration could just as well have been made with a ticket that came out in the opposite direction.

What Happens Across the Dataset

The second block runs the same four prefixes on all 120 records and with four separate seeds. In every row the first two runs, the difference of their mean from the base, the band of the four seeds and the cost per correct label stand side by side.

def measure(p, seed):
    CALL_COUNT[0] = 0
    ti = to = d = 0
    for x in TICKETS:
        c = call(build_prompt(p, x), seed=seed)
        ti, to = ti + c["tokens_in"], to + c["tokens_out"]
        d += read_label(c["response"]) == x["topic"]
    return {"hit_rate": d / len(TICKETS), "total": ti + to}


S = {name: [measure(p, t) for t in SEEDS] for name, p in PREFIXES}
bh = [v["hit_rate"] for v in S["base"]]
base_mean, base_band = (bh[0] + bh[1]) / 2, max(bh) - min(bh)
print(f"{'prompt':<9} {'prefix':>6} {'total':>7} {'run 1':>8} {'run 2':>8} {'effect':>8} "
      f"{'two runs':>10} {'band':>7} {'measurable':>10} {'per correct':>11}")
for name, p in PREFIXES:
    h = [v["hit_rate"] for v in S[name]]
    mean = (h[0] + h[1]) / 2
    effect, gap, band = mean - base_mean, abs(h[0] - h[1]), max(h) - min(h)
    total = S[name][0]["total"]
    limit = max(band, base_band)
    print(f"{name:<9} {len(tokenize(p)):>6d} {total:>7d} {h[0]:>8.4f} {h[1]:>8.4f} {effect:>+8.4f} "
          f"{gap:>10.4f} {band:>7.4f} {('yes' if abs(effect) > limit else 'no'):>10} "
          f"{total / mean / len(TICKETS):>11.1f}")

prompt    prefix   total    run 1    run 2   effect   two runs    band measurable per correct
base           0    5856   0.7500   0.7500  +0.0000     0.0000  0.1500         no        65.1
role A         9    6940   0.7083   0.6583  -0.0667     0.0500  0.0667         no        84.6
role B         9    6942   0.7250   0.6583  -0.0583     0.0667  0.1417         no        83.6
role C        17    7863   0.6417   0.4167  -0.2208     0.2250  0.2250         no       123.8
behavior      27    9105   0.7083   0.8167  +0.0125     0.1083  0.1083         no        99.5

Two Runs Are the Lower Bound of the Band

The first row of the table shows why the criterion was built in this form. The base prompt gave exactly 0.7500 in both runs; the two-run difference is 0.0000. Looking at this number one could have said “this prompt is stable”. When four seeds are run the base gives 0.7500, 0.7500, 0.8583 and 0.7083 and its band comes out at 0.1500. The two runs coinciding is not the absence of uncertainty but two samples falling on the same spot. Two runs are not the measure of the band but its lower bound.

The criterion was made conservative for this reason: an effect that is not larger than the larger of the bands of the two compared prompts counts as unmeasured. None of the four prefixes passes this threshold. The effect of role A is −0.0667 and its threshold 0.1500. The effect of role B is −0.0583 and its threshold 0.1500. The effect of the behavior prefix is +0.0125 and its threshold 0.1500. The effect of role C is −0.2208 and its threshold is its own band, 0.2250. With a looser reading, if only the band of the base had been looked at, role C would pass the threshold, but the direction in which it passes would be down. Whichever reading is chosen the result is the same: none of the four prefixes raised the hit rate in a measurable way.

The only thing measured is the cost, and the cost carries no band. Role A and role B are nine tokens each; 1080 tokens over one hundred twenty requests, and a rise from 5856 to 6940 and 6942 in total. Role C is seventeen tokens, 2040 repeated, total 7863. The behavior prefix is twenty-seven tokens, 3240 repeated, total 9105. The cost per correct label rises from 65.1 tokens to 84.6, 83.6, 123.8 and 99.5 tokens. The prompt that is most expensive per correct label is 1.90 times the base, and it gives nothing measurable in return.

The Role Text Is Not Neutral

The effect of the role prefixes came out unmeasurable, but the direction of their effects is not random. The trace table of the simulator is in the code and is readable. The abone (subscriber) fragment in the text of role A is tied to the tariff label in the trace table; the calis (work) fragment in the text of role C is tied to fault, and the kayit (record) fragment is again tied to fault in the table of misleading cues. So role C adds weight to the fault label in every ticket, independently of the content of the ticket. No fragment in the text of role B appears in either table; its effect is only run noise.

The structural conclusion that follows holds for every text written into a prompt. The role text is not a separate channel; it enters the same token sequence as the task text, and the words it carries mix into the decision. Which words are chosen for a role determines more than how convincing that role is written. While the prompt writer sees the role sentence as style, the model reads it as evidence.

This does not mean that specifying a role is ineffective for every task; this measurement was made on a single task, a single dataset and a single simulator. What it means is narrower and continues throughout the course: a prompt with a role added is not an improved prompt until the difference is measured. Its cost, on the other hand, is known without any measurement, because it is paid for again in every request.

Summary

• In a single ticket the role prefix corrected the answer the base got wrong. The same prefix brought the hit rate over 120 tickets from 0.7500 down to 0.6833. A demonstration over a single ticket presents the movement of the base itself as evidence.
• The base prompt gave 0.7500 in both runs, a two-run difference of 0.0000. Its band across four seeds is 0.1500. Two runs are not the measure of the band but its lower bound.
• Measurability criterion: an effect that is not larger than the larger of the bands of the two compared prompts counts as unmeasured. Role A −0.0667, role B −0.0583, role C −0.2208, behavior +0.0125; all four are below the threshold.
• The cost carries no band and is fully measured: prefixes of 9 to 27 tokens, 1080 to 3240 tokens over 120 requests. The cost per correct label rises from 65.1 to 84.6, 83.6, 123.8 and 99.5 tokens.
• The role text is not a separate channel. The abone fragment is tied to tariff in the trace table, the calis and kayit fragments to fault; the role sentence is read in the same sequence as the task text and mixes into the decision.

Next Step

In this lesson an identity and a procedure were added to the prompt, but what was asked did not change at all: the request was the same sentence in all three prompts. The task definition itself is a design decision, and when it is left ambiguous its cost is larger than that of the prefixes. An ambiguous request does not produce a single wrong answer; it produces a distribution, and that distribution is not visible in a single run. The next lesson measures this: in how many different forms the same ambiguous prompt is answered across ten runs, how far that number falls when the request is made measurable, and how many tokens the narrowing of the distribution costs.