Stating Constraints and Prohibitions

In the previous two lessons everything written into the prompt was positive: do this, write in this format, look at these examples. A large part of the prompts written in practice contains the opposite as well. Do not use a label, do not give a date, do not speak with certainty, do not make things up. These sentences are called negative constraints (prohibitions stated in the prompt), and intuition says that at worst they will have no effect: if the model complies, good; if it does not, nothing changes.

This lesson takes the number of that intuition. The same rule is written in two separate forms, once as what will not be done and once as what will be done, and the number of violations each produces is put side by side. The result is the first example of the second claim of this course: a prompt addition can raise the number instead of lowering it, and there is no way of knowing this without measuring.

• PD54. The model is a simulator, its seed is 20260218, and no real endpoint is called. The support tickets, the response templates and the rule set are fictional.
• PD55. The task is writing a short response: the model assigns the ticket to a label and produces a one-sentence response. The response is built from one of six fictional templates.
• PD56. The rule set is three words: hemen (immediately), yarin (tomorrow) and genel (general) must not appear in the response. They stand for a fictional style guide: do not make promises, do not give dates, do not put vague labels.
• PD57. The violation measure is explicit: if one of these three words appears in the response, the record is in violation. How many responses each word appears in is also counted separately.
• PD58. The simulator chooses the response template according to the tokens that appear in the prompt. Every template has its own distinguishing tokens, and when these tokens appear in the prompt the score of that template rises. The model does not distinguish whether a token is mentioned positively or negatively.
• PD59. Four prompts are compared: no statement, with a negative constraint, with a positive statement, and both together. The task definition, the label set and the ticket set are the same in all four.
• PD60. The cost is written in input tokens, and the unit cost per prevented violation is computed as well.

Two Writings of the Same Rule

# SIMULATOR -- FICTIONAL. This code does not call a real model endpoint; it is a
# language model simulator written with the standard library. The tickets and the response templates are fictional too.
import math

SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW, MAX_OUT = 240, 24


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):                    # FICTIONAL tokenizer: pieces of at most five characters
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def prompt_trace(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
TEMPLATES = {"fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi",
                         "faturami odedim ama sistemde borc gorunuyor",
                         "fatura ayrintisini nereden gorebilirim"],
             "ariza": ["evde su akmiyor bir ariza var galiba",
                       "baglanti bozuk oldu ve hic calismiyor",
                       "ariza kaydi actim ama ekip gelmedi"],
             "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor",
                       "sayac degisimi icin randevu istiyorum",
                       "sayac uzerindeki endeks ile fatura ayni degil"],
             "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum",
                        "tarife degisikligi icin ne yapmam gerekiyor",
                        "birim ucret abone tipine gore degisiyor mu"],
             "kesinti": ["mahallede kesinti var ne zaman gelecek",
                        "planli kesinti duyurusu nereden takip edilir",
                        "aksamdan beri kesinti yasiyoruz"]}
EXTRAS = ["ayrica gecen ay da ayni sorun olmustu", "bu konuda memnun degilim",
          "acil bilgi almam gerekiyor", "kayit numaram elimde yok"]

TICKETS = []
for i in range(120):
    r = rng(SEED + i)
    k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    m = TEMPLATES[k][int(r() * 3)]
    if r() < 0.34:
        m += " " + EXTRAS[int(r() * 4)]
    TICKETS.append({"id": 4001 + i, "text": m, "topic": k})

CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"]
CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura",
        "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza",
        "sayac": "sayac", "endek": "sayac", "okuma": "sayac",
        "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife",
        "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti",
        "borc": "odeme", "odedi": "odeme", "ucret": "odeme",
        "memnu": "sikayet", "sorun": "sikayet", "acil": "sikayet",
        "bilgi": "bilgi", "ogren": "bilgi", "nered": "bilgi"}
TRAPS = {"degis": "tarife", "kayit": "ariza", "yukse": "tarife", "gorun": "fatura",
         "ekip": "kesinti", "takip": "tarife", "giril": "fatura"}
CALL_COUNT = [0]

# FICTIONAL rule set: three words that must not appear in the response.
FORBIDDEN = ["hemen", "yarin", "genel"]
# Every response template comes with a base weight and its own distinguishing tokens.
RESPONSES = [("kaydiniz {k} konusunda acildi , inceleme suruyor", 1.2, ["incel", "acild"]),
             ("{k} talebiniz ilgili ekibe yonlendirildi", 1.1, ["ilgil", "ekibe"]),
             ("bu {k} kaydi hemen ele alinacak", 0.5, ["hemen"]),
             ("{k} kaydiniz yarin sonuclandirilacak", 0.4, ["yarin"]),
             ("talebiniz genel bir basvuru olarak alindi", 0.3, ["genel", "basvu"]),
             ("{k} kaydiniz durumu inceleniyor", 1.0, ["durum", "incel"])]


def call(prompt, seed=SEED, temperature=0.6, max_out=MAX_OUT):
    CALL_COUNT[0] += 1
    tok = tokenize(prompt)
    empty = {"tokens_in": len(tok), "tokens_out": 0, "response": "", "template": -1}
    if len(tok) > WINDOW:
        return dict(empty, status="context_overflow")
    if CALL_COUNT[0] % 37 == 0:
        return dict(empty, status="rate_limit")
    r = rng(seed ^ prompt_trace(prompt))
    cand = [k for k in CONCEPTS if k[:5] in tok]
    if len(cand) < 3:
        cand = CONCEPTS
    p = {k: 0.15 for k in cand}
    for t in tok:
        for table, w in ((CUES, 1.0), (TRAPS, 1.3)):
            if table.get(t) in p:
                p[table[t]] += w
    e = [math.exp(p[a] / temperature) for a in sorted(p)]
    k = sorted(p)[pick(r(), [v / sum(e) for v in e])]
    # Template score: base weight + the number of times the template's tokens appear in the prompt.
    score = [t + 0.85 * sum(a in tok for a in marks) for _, t, marks in RESPONSES]
    q = [math.exp(x / temperature) for x in score]
    i = pick(r(), [x / sum(q) for x in q])
    y = RESPONSES[i][0].format(k=k)
    ot = tokenize(y)
    if len(ot) > max_out:
        ot, y = ot[:max_out], " ".join(ot[:max_out])
    return {"status": "ok", "tokens_in": len(tok), "tokens_out": len(ot),
            "response": y, "template": i}


def read_label(y):
    for s in y.lower().split():
        if s in TOPICS:
            return s
    return None


TASK = ("Talebi su etiketlerden birine ata ve kisa bir yanit yaz : fatura , ariza , "
        "sayac , tarife , kesinti . Talep : ")
PROHIBITION = "Yanitinda su sozcukleri kullanma : hemen , yarin , genel . "
POSITIVE = ("Yanitinda yalnizca sunlar olsun : kaydin acildigi , ilgili ekibe "
            "yonlendirildigi ve durumun inceleniyor olmasi . ")
CASES = (("no statement", ""), ("prohibition", PROHIBITION),
         ("positive", POSITIVE), ("prohibition + positive", PROHIBITION + POSITIVE))


def measure(prefix, seed=SEED):
    CALL_COUNT[0] = 0
    d = {"violation": 0, "labeled": 0, "hit_rate": 0, "tokens_in": 0, "tokens_out": 0}
    counts, templates = {w: 0 for w in FORBIDDEN}, [0] * len(RESPONSES)
    for x in TICKETS:
        s = call(prefix + TASK + x["text"], seed=seed)
        d["tokens_in"], d["tokens_out"] = d["tokens_in"] + s["tokens_in"], d["tokens_out"] + s["tokens_out"]
        if s["template"] >= 0:
            templates[s["template"]] += 1
        found = [w for w in FORBIDDEN if w in s["response"].split()]
        for w in found:
            counts[w] += 1
        d["violation"] += bool(found)
        d["labeled"] += read_label(s["response"]) is not None
        d["hit_rate"] += read_label(s["response"]) == x["topic"]
    for a in ("violation", "labeled", "hit_rate"):
        d[a] /= len(TICKETS)
    return dict(d, counts=counts, templates=templates, calls=CALL_COUNT[0])


print(f"FICTIONAL: {len(TICKETS)} support tickets , forbidden words {FORBIDDEN}")
print(f"\n{'prompt':<23}{'violation':>10}{'labeled':>10}{'hit rate':>10}{'input':>8}"
      f"{'hemen':>7}{'yarin':>7}{'genel':>7}")
O = {}
for name, prefix in CASES:
    d = O[name] = measure(prefix)
    print(f"{name:<23}{d['violation']:>10.4f}{d['labeled']:>10.4f}{d['hit_rate']:>10.4f}"
          f"{d['tokens_in']:>8}" + "".join(f"{d['counts'][w]:>7}" for w in FORBIDDEN))

FICTIONAL: 120 support tickets , forbidden words ['hemen', 'yarin', 'genel']

prompt                  violation   labeled  hit rate   input  hemen  yarin  genel
no statement               0.2583    0.9000    0.6667    5060      5     17      9
prohibition                0.5333    0.8583    0.5750    6740     35     16     13
positive                   0.0167    0.9667    0.8333    8420      1      0      1
prohibition + positive     0.1583    0.9417    0.6500   10100     12      3      4

The first row is the baseline: with no statement, a share of 0.2583 of the responses carries one of the forbidden words. The second row is the number of this lesson. When the negative constraint is added the violation rate becomes 0.5333, that is, more than double. Writing the prohibition does not prohibit the forbidden thing; it doubles it. The three columns on the right show where the increase gathers: the number of responses carrying the word hemen goes from 5 to 35, genel goes from 9 to 13 and yarin goes from 17 to 16. The increase is not evenly distributed, and the only fall is one unit.

The third row is the positive writing of the same rule. The prompt never mentions which words are forbidden; instead it says what will be in the response. The violation rate becomes 0.0167, one fifteenth of the baseline. The side effect is also in the positive direction: the share of responses with a readable label goes from 0.9000 to 0.9667 and the label hit rate goes from 0.6667 to 0.8333. The same rule, the same model, the same task and the same set; the only thing that changes is the sentence of the rule.

The fourth row is the writing most often seen in practice: first what is wanted is said, then “and do not do these” is added. The violation rate becomes 0.1583, about ten times the 0.0167 that the positive statement gives alone. When the prohibition sentence is added next to a positive prompt it carries its own damage as well. The label hit rate also falls back from 0.8333 to 0.6500.

Why the Prohibition Works in Reverse

print("template                           none prohibit positive     both")
for i, (text, _, marks) in enumerate(RESPONSES):
    label = ("forbidden: " + [w for w in FORBIDDEN if w in text][0]) if any(
        w in text for w in FORBIDDEN) else "allowed: " + marks[0]
    print(f"{label:<30}" + "".join(f"{O[name]['templates'][i]:>9}" for name, _ in CASES))

print("\nrobustness of the increase across runs")
for sd in (SEED, SEED + 1, SEED + 2):
    v = [measure(prefix, seed=sd)["violation"] for _, prefix in CASES]
    print(f"  seed {sd}: " + " ".join(f"{x:.4f}" for x in v))
v0 = [measure(prefix)["violation"] for _, prefix in CASES]
v1 = [measure(prefix, seed=SEED + 1)["violation"] for _, prefix in CASES]
print("  two-run difference: " + " ".join(f"{abs(a - b):.4f}" for a, b in zip(v0, v1)))

n = len(TICKETS)
base = O["no statement"]
print("\ncost")
for name, _ in CASES[1:]:
    d, diff = O[name], round((O[name]["violation"] - base["violation"]) * n)
    extra = d["tokens_in"] - base["tokens_in"]
    unit = f"{extra / abs(diff):.1f}" if diff else "-"
    print(f"  {name:<22} violations {round(base['violation'] * n):>3} -> {round(d['violation'] * n):>3}"
          f" ({diff:+d}) , input tokens {extra:+d} , per violation {unit}")

template                           none prohibit positive     both
allowed: incel                       39       18       19       52
allowed: ilgil                       26       16       77       23
forbidden: hemen                      5       35        1       12
forbidden: yarin                     17       16        0        3
forbidden: genel                      9       13        1        4
allowed: durum                       21       19       19       23

robustness of the increase across runs
  seed 20260218: 0.2583 0.5333 0.0167 0.1583
  seed 20260219: 0.2750 0.4167 0.0000 0.1167
  seed 20260220: 0.3833 0.4917 0.0083 0.1167
  two-run difference: 0.0167 0.1167 0.0167 0.0417

cost
  prohibition            violations  31 ->  64 (+33) , input tokens +1680 , per violation 50.9
  positive               violations  31 ->   2 (-29) , input tokens +3360 , per violation 115.9
  prohibition + positive violations  31 ->  19 (-12) , input tokens +5040 , per violation 420.0

The chart above makes the mechanism clear. In the prompt with no statement the hemen template is chosen 5 times; in the prompt with the negative constraint 35 times. The prohibition sentence wrote the token of this template’s text into the prompt, and the simulator processed that token as a word it saw in the prompt. The model does not distinguish whether a word is mentioned together with “do not use” or together with “use”; what it sees is the presence of the word. The positive statement uses the same mechanism in the opposite direction: the allowed template with the ilgil token goes from 26 to 77, because this time the words that appear in the prompt are the words of the allowed template.

The middle block tests whether the difference is independent of the run. The ordering is the same in all three seeds: the negative constraint is above the baseline, and the positive statement is far below the baseline. The difference between two runs is 0.0167 in the prompt with no statement, 0.0167 in the positive statement and 0.1167 in the negative constraint. The +0.2750 that the negative constraint adds is larger than twice its own run difference and counts as measured. The -0.2416 that the positive statement adds is fourteen times the run difference and is beyond dispute. That the run difference of the negative constraint is seven times larger than the other two is a finding in itself: the prohibition does not only worsen the result, it makes it more unstable too.

The sub-block gives the cost. The negative constraint adds 1680 input tokens and raises the number of violations from 31 to 64; every token added pushes the number in the wrong direction. The positive statement adds 3360 input tokens and brings the violations from 31 down to 2: 115.9 input tokens per prevented violation. Adding the prohibition to the positive statement spends 5040 tokens and prevents only 12 violations, that is, 420.0 tokens per prevented violation, when there is a prompt that does the same job at 115.9 tokens per prevented violation. The prompt that spends twice as many tokens is the right prompt here, and it is possible to say this only by running both and counting. A prompt changed without measuring is not an improvement but a guess; in this lesson the guess came out in the opposite direction of the number.

Summary

• The same rule was measured with two separate writings. The violation rate is 0.2583 in the prompt with no statement and 0.5333 in the prompt with the negative constraint: writing the prohibition raised the violation to more than double.
• The increase gathers in the forbidden words: the number of responses carrying hemen goes from 5 to 35, genel goes from 9 to 13 and yarin goes from 17 to 16.
• The positive statement builds the same rule without mentioning any forbidden word and brings the violation down to 0.0167; the label hit rate goes from 0.6667 to 0.8333.
• When both are written together the violation becomes 0.1583, about ten times what the positive statement gives alone. The prohibition sentence carries its own damage inside a positive prompt too.
• The cost: the negative constraint adds 33 violations with +1680 input tokens, the positive statement prevents 29 violations with +3360 input tokens (115.9 tokens per prevented violation), and both together prevent only 12 violations with +5040 tokens.

Next Step

In four lessons four separate design decisions were measured, and each was tried alone, in a single shot. A real prompt is not written this way. A prompt is written, measured, a part of it is changed, measured again, and this loop continues until the number stops improving. The next lesson puts the loop itself under measurement: in how many iterations the gain saturates and, the more disturbing question, when the number of the set looked at during the iterations and the number of a set never looked at begin to separate. This lesson is also the closing of the topic and ties the total of the eight lessons together with a question.