Penalties and Stop Sequences

Last lesson the sampling settings controlled which concept would be chosen. Where the response would end was left to a single hard length limit, and how many responses were truncated at that limit was never counted. In the same way, a long response repeating the same sentence within itself was not measured either.

This lesson adds two controls and compares both with numbers. The repetition penalty lowers the score of a piece that has been produced before. The stop sequence ends the generation when a given piece of text is produced. Both are call settings and neither adds a token to the prompt; so the lesson also measures a third path: asking for the same thing by writing it into the prompt.

• OC9. The code is a simulator; no real endpoint is called. The tickets and the response sentences are fictional, seed 20260218.
• OC10. The simulator produces the response sentence by sentence. At each step one of six fictional sentences is drawn according to its score; if the draw is unconstrained, the same sentence can come up more than once.
• OC11. Repetition rate is the share of repeated sentences in a response: one minus the ratio of the number of distinct sentences to the total number of sentences, averaged over 120 responses.
• OC12. Truncated response is a response that hits the output token limit (34 tokens) and has its last sentence left half-written. Prematurely cut response is a response that ends without the label sentence being written because of the stop sequence; the two are counted separately.
• OC13. The label sentence is the sentence that states the topic was recorded. A last sentence left half-written does not count as complete; the usability of a response is measured by looking at this sentence.
• OC14. The negative constraint in the prompt corresponds to a mild penalty in the simulator. This is a fictional assumption showing that an instruction is not directly translated into a setting.
• OC15. Two runs are taken with seeds 20260218 and 20260219; the repetition rate is printed separately for the two runs and the difference between them is the measurability threshold.
• OC16. The cost is written as input tokens, output tokens and the number of calls; the number of calls never changes in this lesson.

A Response Repeating Itself

A response is not a single choice; it is many choices one after another. At each step the highest-scored piece is a little more probable, and having been chosen once does not make it less probable. The result is a loop that shows itself in long responses: the same sentence keeps coming back until the limit truncates the response.

# SIMULATOR -- FICTIONAL. No real model endpoint is called. This block is a language model
# simulator that produces the response sentence by sentence; the tickets are FICTIONAL too.
import math

SEED, M32 = 20260218, 0xFFFFFFFF
MAX_OUT, MAX_SENTENCES = 34, 6          # output token limit and sentence limit


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(text):                     # FICTIONAL tokenizer, a piece has at most five characters
    out = []
    for piece in text.lower().replace(",", " , ").split():
        while len(piece) > 5:
            out.append(piece[:5])
            piece = piece[5:]
        out.append(piece)
    return out


def trace_id(text):
    h = 2166136261
    for c in text:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"),
            ("faturami odedim ama sistemde borc gorunuyor", "fatura"),
            ("fatura ayrintisini nereden gorebilirim", "fatura"),
            ("evde su akmiyor bir ariza var galiba", "ariza"),
            ("baglanti bozuk oldu ve hic calismiyor", "ariza"),
            ("ariza kaydi actim ama ekip gelmedi", "ariza"),
            ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"),
            ("sayac degisimi icin randevu istiyorum", "sayac"),
            ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"),
            ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"),
            ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"),
            ("birim ucret abone tipine gore degisiyor mu", "tarife"),
            ("mahallede kesinti var ne zaman gelecek", "kesinti"),
            ("planli kesinti duyurusu nereden takip edilir", "kesinti"),
            ("aksamdan beri kesinti yasiyoruz", "kesinti")]
TICKETS = []
for i in range(120):                    # 120 FICTIONAL support tickets
    r = rng(SEED + i)
    topic = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    TICKETS.append({"id": 4001 + i, "topic": topic,
                    "text": [x for x in PATTERNS if x[1] == topic][int(r() * 3)][0]})

CUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza",
        "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac",
        "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife",
        "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti",
        "mahal": "kesinti"}
SENTENCE_BANK = [("talebiniz alindi", 1.00), ("konu {k} olarak kaydedildi", 1.25),
                 ("ilgili ekibe yonlendirildi", 1.05), ("en kisa surede donus yapilacak", 0.95),
                 ("ek bilgi icin kayit numaranizi hazir bulundurun", 0.80),
                 ("anlayisiniz icin tesekkur ederiz", 0.85)]
COUNTER = [0]


def respond(prompt, seed=SEED, penalty=0.0, stop=None, max_out=MAX_OUT):
    """The response is produced sentence by sentence. The penalty lowers the score of a sentence
    that was used; if the stop sequence appears in a produced sentence, the generation ends
    without writing that sentence."""
    COUNTER[0] += 1
    toks = tokenize(prompt)
    p = {k: 0.15 for k in TOPICS}
    for t in toks:
        if CUES.get(t) in p:
            p[CUES[t]] += 1.0
    k = max(sorted(p), key=lambda a: p[a])
    if "tekra" in toks:                                 # negative constraint in the prompt
        penalty = max(penalty, 0.45)
    r = rng(seed ^ trace_id(prompt))
    used, sentences, emitted, truncated, early = {}, [], [], False, False
    while len(sentences) < MAX_SENTENCES:
        s = [b - penalty * used.get(c, 0) for c, b in SENTENCE_BANK]
        e = [math.exp(v / 0.35) for v in s]
        c = SENTENCE_BANK[pick(r(), [v / sum(e) for v in e])][0].format(k=k)
        if stop and stop in c:                          # stop sequence
            early = not any("konu" in x for x in sentences)
            break
        used[c.replace(k, "{k}")] = used.get(c.replace(k, "{k}"), 0) + 1
        new = tokenize(c)
        if len(emitted) + len(new) > max_out:           # length limit: truncation
            emitted, truncated = (emitted + new)[:max_out], True
            sentences.append(c)
            break
        emitted += new
        sentences.append(c)
    complete = sentences[:-1] if truncated else sentences
    return {"sentences": sentences, "tokens": len(emitted), "truncated": truncated,
            "early": early, "label": any(c.startswith("konu") for c in complete)}


INSTRUCTION = "Su destek talebine kisa bir yanit yaz . Talep "
SAMPLE = TICKETS[4]
o = respond(INSTRUCTION + f"{SAMPLE['id']} : {SAMPLE['text']}")
print(f"FICTIONAL ticket set {len(TICKETS)} records; sample ticket {SAMPLE['id']}")
print(f"output limit {MAX_OUT} tokens, sentence limit {MAX_SENTENCES}")
for c in o["sentences"]:
    print(f"  - {c}")
print(f"sentences {len(o['sentences'])}, distinct sentences {len(set(o['sentences']))}, "
      f"tokens {o['tokens']}, truncated {o['truncated']}, label {o['label']}")
d = respond(INSTRUCTION + f"{SAMPLE['id']} : {SAMPLE['text']}", penalty=0.90,
            stop="ek bilgi")
print("same ticket with penalty 0.90 and stop sequence 'ek bilgi':")
for c in d["sentences"]:
    print(f"  - {c}")
print(f"sentences {len(d['sentences'])}, distinct sentences {len(set(d['sentences']))}, "
      f"tokens {d['tokens']}, truncated {d['truncated']}, early {d['early']}")

FICTIONAL ticket set 120 records; sample ticket 4005
output limit 34 tokens, sentence limit 6
  - anlayisiniz icin tesekkur ederiz
  - ilgili ekibe yonlendirildi
  - en kisa surede donus yapilacak
  - talebiniz alindi
  - ilgili ekibe yonlendirildi
  - en kisa surede donus yapilacak
sentences 6, distinct sentences 4, tokens 34, truncated True, label False
same ticket with penalty 0.90 and stop sequence 'ek bilgi':
  - anlayisiniz icin tesekkur ederiz
  - ilgili ekibe yonlendirildi
  - en kisa surede donus yapilacak
  - talebiniz alindi
  - konu sayac olarak kaydedildi
sentences 5, distinct sentences 5, tokens 31, truncated False, early False

The same ticket received two different responses under two settings, and the difference is not a matter of style. The upper response carries two repetitions, with four of six sentences distinct, spends all thirty-four tokens, is truncated at the limit and never manages to write the label sentence. The response is unusable: the assigned topic is not in it. The lower response ends at thirty-one tokens with five distinct sentences, and the label sentence makes it to the end. No truncation, no repetition.

The causality matters here. Repetition does not only make the response ugly; it uses up the limited budget and leaves the actual content out. When repetition combines with the length limit, it stops being a formatting problem and becomes a loss of content.

Counting the Three Paths in the Same Table

The same 120 tickets are run under six scaffoldings. The second row is a negative constraint written into the prompt; the third and fourth rows are the repetition penalty, which is a call setting; the fifth row is the stop sequence, and the sixth is the combination of the two.

NEGATIVE_CONSTRAINT = "Ayni cumleyi tekrar etme . "
SCAFFOLDING = [("base", INSTRUCTION, 0.0, None),
               ("negative constraint in prompt", NEGATIVE_CONSTRAINT + INSTRUCTION, 0.0, None),
               ("repetition penalty 0.45", INSTRUCTION, 0.45, None),
               ("repetition penalty 0.90", INSTRUCTION, 0.90, None),
               ("stop sequence", INSTRUCTION, 0.0, "ek bilgi"),
               ("penalty + stop sequence", INSTRUCTION, 0.90, "ek bilgi")]


def measure(instruction, penalty, stop, seed=SEED):
    COUNTER[0] = 0
    rep = trunc = early = usable = n_in = n_out = 0
    for x in TICKETS:
        prompt = instruction + f"{x['id']} : {x['text']}"
        o = respond(prompt, seed=seed, penalty=penalty, stop=stop)
        n = len(o["sentences"])
        rep += 0.0 if n == 0 else 1 - len(set(o["sentences"])) / n
        trunc += o["truncated"]
        early += o["early"]
        usable += o["label"]
        n_in += len(tokenize(prompt))
        n_out += o["tokens"]
    return (rep / len(TICKETS), trunc, early, usable / len(TICKETS), n_in, n_out)


print("scaffolding                    repetition A  repetition B  truncated  early  has label  input  output")
for name, instruction, pen, stop in SCAFFOLDING:
    a = measure(instruction, pen, stop)
    b = measure(instruction, pen, stop, SEED + 1)
    print(f"{name:<30} {a[0]:>12.4f}  {b[0]:>12.4f}  {a[1]:>4}/{b[1]:<4}  {a[2]:>5}"
          f"  {a[3]:>9.4f}  {a[4]:>5}  {a[5]:>6}")

scaffolding                    repetition A  repetition B  truncated  early  has label  input  output
base                                 0.3431        0.3750   107/106       0     0.8833   2734    4073
negative constraint in prompt        0.2167        0.1939   112/117       0     0.9083   3574    4072
repetition penalty 0.45              0.1972        0.1944   117/117       0     0.9667   2734    4078
repetition penalty 0.90              0.0892        0.0925   120/120       0     0.9833   2734    4080
stop sequence                        0.2685        0.3228    55/59       25     0.7417   2734    2942
penalty + stop sequence              0.0578        0.0715    24/25       20     0.8333   2734    2653

The second and third rows are the real comparison of this lesson. Both ask for the same thing: do not repeat yourself. The negative constraint lowers the repetition rate from 0.3431 to 0.2167 and raises the input tokens from 2734 to 3574 — seven tokens per ticket, 840 in total. The repetition penalty lowers the same rate to 0.1972 and the input tokens stay constant at 2734. The less effective path is expensive, the more effective path is free. That a target can be written into the prompt does not mean that writing it there is right.

When the penalty is raised from 0.45 to 0.90, the repetition falls to 0.0892 and the gain is 0.1080. On this row the two-run difference is only 0.0033, so the gain is more than thirty times the difference: an improvement that is comfortably measurable. The same reading gives the opposite result on the fifth row. The stop sequence lowers the repetition from 0.3431 to 0.2685, a gain of 0.0746; but the two-run difference of that row is 0.0543. The gain stays only a little above the difference; a single run would make this row look stronger than it is. A prompt changed without being measured is not an improvement but a guess, and measurement requires reading the two runs together.

That the two-run difference changes from row to row is also information. On the penalty rows it stays between 0.0033 and 0.0028, because the penalty prevents a second choice of the same sentence to the same extent in every run. On the stop sequence row the difference goes up to 0.0543, because where the response ends depends on a single draw and that draw shifts with the seed. While a scaffolding improves the number it measures, it can lower the reliability of that number; a table that does not print two runs does not show this.

The table also shows that while the repetition rate falls, the number of truncated responses rises: 107 on the base, 120 with penalty 0.90. The penalty reduces repetition but does not shorten the response; on the contrary, the distinct sentences that come in place of the repetition are longer and use up the budget faster. The only scaffolding that solves the length problem is the stop sequence: the truncated responses fall to 55, and to 24 together with the penalty, and the output tokens drop from 4073 to 2653. The total gain is 1420 tokens, 11.8 per response.

What the Stop Sequence Cuts Early

The cost of the stop sequence is paid not in the output but in the content. In the fifth row, twenty-five responses end without the label sentence being written at all: the sentence carrying the stop sequence fell early in the order and the generation stopped there. The share of usable responses falls from the base’s 0.8833 to 0.7417. That is, the 1131 tokens gained from the output tokens were paid for with seventeen usable responses.

The last row shows how the two scaffoldings correct each other. When the repetition penalty is added, the early cuts go from 25 to 20 and the usability rises to 0.8333. The reason is the order: the penalty lowers the score of sentences that were used before, so the label sentence moves toward the beginning of the response and in most responses it has already been written when the stop sequence arrives. Even so, the last row is still below the base: 0.8333 against 0.8833. Together the two scaffoldings made the output a third cheaper and lowered the usability.

This shows that choosing the stop sequence is a design decision. The sequence is not arbitrary: a piece of text expected to appear after the mandatory content in the response must be chosen. The piece chosen here was the beginning of an additional-information sentence expected near the end of the response, and in the twenty-five responses where the expectation did not hold, content was lost. What needs to be measured is not the sequence itself but how often it appears before the mandatory content; without printing this number the stop sequence does not count as measured.

To wrap up: the base prompt is the plain response instruction: 2734 input tokens, repetition 0.3431, truncated 107, output 4073, usability 0.8833. The changed prompt can be built in two separate ways. Writing a negative constraint into the prompt lowers the repetition to 0.2167 and adds 840 tokens. Changing the call setting lowers the repetition to 0.0578, the truncated responses to 24 and the output to 2653; the input tokens and the number of calls never change, in return twenty responses are cut early and the usability falls below the base. The cost of a scaffolding is not always paid in tokens; sometimes it is paid directly in usability.

Summary

• Because the response is produced sentence by sentence, repetition arises by itself: on the base scaffolding the repetition rate is 0.3431 and 107 of 120 responses are truncated at the length limit.
• Repetition is not a style problem; it uses up the limited output budget, leaves the label sentence out and makes the response unusable.
• A negative constraint written into the prompt lowers the repetition to 0.2167 and adds 840 input tokens; the 0.45 repetition penalty, which is a call setting, lowers the same rate to 0.1972 and adds zero tokens.
• When the penalty goes from 0.45 to 0.90 the gain is 0.1080 and the two-run difference is 0.0033; the stop sequence’s gain over the base is 0.0746 and the two-run difference is 0.0543, that is, the second row is marginal.
• The stop sequence lowers the truncated responses from 107 to 55, and to 24 together with the penalty, and brings the output tokens from 4073 to 2653; its cost is twenty-five prematurely cut responses and usability falling from 0.8833 to 0.7417.

Next Step

These two controls arranged how long the response lasted, not what it looked like. Truncation decreased, repetition decreased, but the output is still free text: in which sentence of the response the label appears, in which word it is written and what else is next to it is different every time. An output that will be written into a record system cannot be read in this form. The next lesson compares the format of the output on three levels — free text, a format description written into the prompt and an imposed schema — and at each level prints the rate of responses that match the schema together with the token cost.