Message-Based Interfaces

The Prompt Engineering course turned the prompt into a design object over twenty-two lessons and wrote the cost of every change in tokens. Every line of it carried a silent assumption: the call happened alone, instantly and without error.

None of these assumptions holds inside an application. A call travels over a network, queues, gets rejected, gets cut off midway and each time spends tokens instead of money. This is the rule of this course: a model call is not a function call but a network operation; a call whose failure is not written down counts as unmeasured. The first lesson starts from the most basic property of the interface. In a message-based interface conversation state is not kept on the server; the transcript lives on the client and is resent with every request. This lesson counts what that resending costs and how far five send policies bring it down.

• AU1. The code is a simulator; no real model endpoint is called. The conversations, request texts and label set are fictional, seed 20260218.
• AU2. The simulator also carries network behavior: rate limit rejection (every thirty-seventh call), context window overflow, timeout and mid-response cut of the response. The thresholds are fictional; what is measured is not the value of the threshold but the client’s response to it.
• AU3. The interface recognizes three roles: system, user and model. Role names also count as tokens.
• AU4. Conversation state is not kept on the server. The transcript stays on the client; what to send with each request is a client decision, and this lesson measures five decisions.
• AU5. Latency is written not as raw time but as steps: the first token step depends on the input, the total steps on the output. A conversation is eight sequential calls; the latency class is the chain.
• AU6. The fictional conversation has eight turns. The key turn that determines the topic falls between the second and the sixth; the remaining turns are filler and three of them carry a misleading word.
• AU7. The metric is the label given in the last turn. A call that is rejected, times out or whose label is not completed counts as wrong.
• AU8. The cost is input tokens, output tokens, steps and failure share; the number of calls is 320 in all five modes.

Message-Based Interface and a Simulator with Network Behavior

Resource layout, versioning, the error response format and the contract were measured in the Web API Design course; that procedure is not repeated. The only thing a message-based interface adds is the shape of the body: the request is not a set of fields but a list of role and content pairs.

# SIMULATOR -- FICTIONAL. No real endpoint is called; the network layer is simulated too:
# rate limit rejection, mid-response cut, chunk-by-chunk generation.
SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW, MAX_LENGTH, RATE_EVERY = 240, 24, 37


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def split_pieces(m):
    """FICTIONAL tokenizer; a piece has at most five characters."""
    b = []
    for s in m.lower().replace(",", " , ").split():
        new = True
        while len(s) > 5:
            b.append((s[:5], new))
            s, new = s[5:], False
        b.append((s, new))
    return b


def tokenize(m):
    return [p for p, _ in split_pieces(m)]


def join_pieces(pl):
    """Binds the pieces back into words."""
    s = ""
    for p, new in pl:
        s += (" " if new and s else "") + p
    return s


def hash_text(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


def draw(prompt, seed, salt):
    return rng((seed + salt) ^ hash_text(prompt))()


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
CONCEPTS = TOPICS + ["odeme", "sikayet", "bilgi", "genel"]
CLUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "borc": "fatura",
         "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza",
         "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "rande": "sayac",
         "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife",
         "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti",
         "odeme": "odeme", "sikay": "sikayet", "bilgi": "bilgi"}
TRAPS = {"degis": "tarife", "kayit": "ariza", "ekip": "kesinti", "takip": "tarife"}
ACK = ["talebinizi aldim ve kaydi guncelledim",     # when no label is requested
       "ilgili ekibe ilettim donus yapilacak",
       "kaydinizi kontrol ediyorum lutfen bekleyin"]
CALL_COUNT = [0]


def flatten(messages):
    """Role labels count as tokens too."""
    return " ".join(f"{r} : {i}" for r, i in messages)


def call(messages, seed=SEED, temperature=0.6, max_length=MAX_LENGTH):
    """Message-based interface. messages: [(role, content)], role sistem|kullanici|model."""
    CALL_COUNT[0] += 1
    prompt = flatten(messages)
    toks = tokenize(prompt)
    first = 1 + len(toks) // 40                   # steps until the first token
    empty = {"input": len(toks), "output": 0, "response": "", "first": first, "steps": first}
    if len(toks) > WINDOW:
        return dict(empty, status="context_overflow")
    if CALL_COUNT[0] % RATE_EVERY == 0:           # rate limit by call count
        return dict(empty, status="rate_limit")
    r = rng(seed ^ hash_text(prompt))
    if len(toks) > 90 and r() < 0.04:
        return dict(empty, status="timeout")
    candidates = [k for k in CONCEPTS if k[:5] in toks]
    if len(candidates) < 3:                       # the label set narrows the candidates
        candidates = list(CONCEPTS)
    p = {k: 0.15 for k in candidates}
    for t in toks:
        for table, w in ((CLUES, 1.0), (TRAPS, 0.6)):
            if table.get(t) in p:
                p[table[t]] += w
    names = sorted(p)
    if temperature <= 0.0:
        k = max(names, key=lambda a: p[a])
    else:
        from math import exp
        e = [exp(p[a] / temperature) for a in names]
        k = names[pick(r(), [x / sum(e) for x in e])]
    c = draw(prompt, seed, 17)
    if "tek" in toks:                             # only the label was requested
        text = k
    else:
        text = ACK[pick(c, [0.40, 0.33, 0.27])]
    pl, status = split_pieces(text), "done"
    if len(pl) > max_length:                      # stop condition
        pl, status = pl[:max_length], "stopped"
    u = draw(prompt, seed, 91)                    # the network cuts the response midway
    if len(pl) > 3 and u < 0.08:
        pl, status = pl[:1 + int(u / 0.08 * (len(pl) - 1))], "cut"
    return {"status": status, "input": len(toks), "output": len(pl),
            "response": join_pieces(pl), "first": first, "steps": first + len(pl)}


def read_label(y):
    return next((s for s in y.split() if s in TOPICS), None)


OPENERS = ["merhaba yardim gerekiyor", "iyi gunler destek istiyorum",
           "selam bir sorunum var", "merhaba destek almak istiyorum"]
KEY_TURNS = {"fatura": ["bu ayki fatura tutari gecen aya gore cok yuksek geldi",
                        "faturami odedim ama sistemde borc gorunuyor",
                        "fatura ayrintisini nereden gorebilirim"],
             "ariza": ["evde su akmiyor bir ariza var galiba",
                       "baglanti bozuk oldu ve hic calismiyor",
                       "ariza kaydi actim ama ekip gelmedi"],
             "sayac": ["sayac okumasi yanlis girilmis endeks tutmuyor",
                       "sayac degisimi icin randevu istiyorum",
                       "sayac uzerindeki endeks ile fatura ayni degil"],
             "tarife": ["hangi tarife basamagindayim ogrenmek istiyorum",
                        "tarife degisikligi icin ne yapmam gerekiyor",
                        "birim ucret abone tipine gore degisiyor mu"],
             "kesinti": ["mahallede kesinti var ne zaman gelecek",
                         "planli kesinti duyurusu nereden takip edilir",
                         "aksamdan beri kesinti yasiyoruz"]}
FILLER = ["tesekkur ederim not aldim", "peki bunun icin ne kadar beklemem gerekir",
          "daha once de ayni durum olmustu", "yazili bir bildirim alabilir miyim",
          "bu konuda kayit acildi mi", "degisiklik yapilirsa haber verilir mi",
          "ekip ne zaman gelir", "hafta sonu da ulasabilir miyim"]
SYSTEM = ("sistem", "destek yardimcisisin . kisa yanit ver .")
QUESTION = ("gorusme {n} konusunu su etiketlerden birine ata : fatura , ariza , "
            "sayac , tarife , kesinti . tek etiket yaz")

CONVERSATIONS = []
for i in range(40):                               # 40 FICTIONAL conversations, eight turns each
    r = rng(SEED + 7 * i)
    k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    key_text = KEY_TURNS[k][int(r() * 3)]
    key_pos = 1 + int(r() * 5)                    # key turn from two to six
    turns = [OPENERS[int(r() * 4)]]
    for t in range(1, 7):
        turns.append(key_text if t == key_pos else FILLER[int(r() * 8)])
    turns.append(QUESTION.format(n=7301 + i))
    CONVERSATIONS.append({"id": 7301 + i, "topic": k, "key_pos": key_pos + 1, "turns": turns})

g = CONVERSATIONS[0]
print(f"conversation {g['id']} -- topic {g['topic']} , key turn {g['key_pos']}")
history, sent = [SYSTEM], []
for t, text in enumerate(g["turns"], 1):
    history.append(("kullanici", text))
    s = call(history)
    sent.append(s["input"])
    if t in (1, g["key_pos"], 8):
        print(f"  turn {t}  sent {s['input']:>3}  output {s['output']:>2}"
              f"  first token {s['first']}  {s['status']:<6} -> {s['response'][:34]}")
    history.append(("model", s["response"] or "-"))
print("  input tokens sent per turn:", sent)
print(f"  total {sum(sent)} tokens , {sum(sent) / sent[-1]:.2f} times the last turn")
print("  same last turn with seed 20260219 ->", call(history[:-1], SEED + 1)["response"])

conversation 7301 -- topic kesinti , key turn 5
  turn 1  sent  21  output  8  first token 1  done   -> talebinizi aldim ve kaydi guncelle
  turn 5  sent  99  output 10  first token 3  done   -> kaydinizi kontrol ediyorum lutfen 
  turn 8  sent 187  output  2  first token 5  done   -> kesinti
  input tokens sent per turn: [21, 41, 53, 75, 99, 123, 142, 187]
  total 741 tokens , 3.96 times the last turn
  same last turn with seed 20260219 -> kesinti

The user wrote eight short sentences, but the total going to the interface is 741 tokens, which is 3.96 times what the last turn costs alone. The reason is plain: every turn carries all the turns before it again. If turn lengths are roughly equal, the total sent over an $n$-turn conversation grows as $n(n+1)/2$; when the number of turns doubles, the cost approaches fourfold. The first token step also rises from 1 to 5: a growing state is not only expensive but slow. In the last line the same last turn with the second seed again returned outage; in a single call the output stayed fixed, but the table below will show that this stability does not hold at the scale of forty conversations.

Five Send Policies

Since the transcript is on the client, which part of it to send with each request is a design decision. Five decisions run on the same forty conversations: sending no state, the whole transcript, the last three turns, the user turns only, and an extracted state record produced on the client instead of the transcript. The extractor of the last one recognizes seven fields and silently drops what it does not recognize.

FIELDS = {"fatur": "fatura", "borc": "borc", "ariza": "ariza", "sayac": "sayac",
          "endek": "endeks", "tarif": "tarife", "kesin": "kesinti"}


def state_record(history):
    """State kept on the client: not the transcript, a list of fields."""
    a = []
    for role, content in history:
        if role == "kullanici":
            for t in tokenize(content):
                if t in FIELDS and FIELDS[t] not in a:
                    a.append(FIELDS[t])
    return " , ".join(a) if a else "yok"


def send(mode, history, now):
    """Decision on which part of the transcript to send."""
    current = [("kullanici", now)]
    if mode == "stateless":
        return [SYSTEM] + current
    if mode == "full state":
        return [SYSTEM] + history + current
    if mode == "last three turns":
        return [SYSTEM] + history[-6:] + current
    if mode == "user only":
        return [SYSTEM] + [m for m in history if m[0] == "kullanici"] + current
    return [SYSTEM, ("kullanici", "durum : " + state_record(history))] + current


def measure(mode, seed=SEED):
    CALL_COUNT[0] = 0
    tok_in = tok_out = steps = correct = failed = 0
    for conv in CONVERSATIONS:
        history = []
        for t, text in enumerate(conv["turns"], 1):
            s = call(send(mode, history, text), seed)
            tok_in, tok_out, steps = tok_in + s["input"], tok_out + s["output"], steps + s["steps"]
            failed += s["status"] != "done"
            history += [("kullanici", text), ("model", s["response"] or "-")]
            if t == 8:
                correct += read_label(s["response"]) == conv["topic"]
    return tok_in, tok_out, steps, correct, failed / 320


MODES = ["stateless", "full state", "last three turns", "user only", "state record"]
print("mode               input output steps  accuracy A  accuracy B    failure"
      "   per correct")
for mode in MODES:
    a, b = measure(mode), measure(mode, SEED + 1)
    print(f"{mode:<17} {a[0]:>6} {a[1]:>6} {a[2]:>5}  {a[3] / 40:>10.4f}"
          f"  {b[3] / 40:>10.4f}  {a[4]:>9.4f}  {(a[0] + a[1]) / a[3]:>12.1f}")

g0 = CONVERSATIONS[0]
print("\nstate record , conversation", g0["id"], ":",
      state_record([("kullanici", m) for m in g0["turns"][:7]]))
no_field = sum(state_record([("kullanici", m) for m in conv["turns"][:7]]) == "yok"
            for conv in CONVERSATIONS)
print("conversations where the extractor found no field:", no_field, "/ 40")

history, t = [], 0
while True:                                  # where the full state overflows the window
    t += 1
    text = g0["turns"][t - 1] if t <= 7 else FILLER[t % 8]
    s = call([SYSTEM] + history + [("kullanici", text)])
    if s["status"] == "context_overflow":
        print(f"full state exceeds the window at turn {t}: {s['input']} > {WINDOW}")
        break
    history += [("kullanici", text), ("model", s["response"])]

mode               input output steps  accuracy A  accuracy B    failure   per correct
stateless           8346   2350  2710      0.2000      0.2500     0.0406        1337.0
full state         31160   2341  3294      0.8250      0.7000     0.0625        1015.2
last three turns   23102   2347  3132      0.4500      0.4250     0.0844        1413.8
user only          19738   2297  2946      0.6750      0.7750     0.1000         816.1
state record       10474   2393  2753      0.6500      0.5500     0.0375         494.9

state record , conversation 7301 : kesinti
conversations where the extractor found no field: 5 / 40
full state exceeds the window at turn 12: 246 > 240

The first row is the client without scaffolding: when no state is carried, the last turn sees only the label set and the conversation number, and the accuracy of 0.2000 is the chance share among five labels. The second row does the same job with conversation state and raises the accuracy to 0.8250. The cost is in the input column: from 8346 to 31160, that is 3.73 times. Steps go from 2710 to 3294, because the growing prompt delays the first token; the failure share also rises from 0.0406 to 0.0625.

Measured Gain, Unmeasured Gain and Silent Loss

The three shortening decisions split the table down the middle. Sending the last three turns cuts the input by 26% but drops the accuracy to 0.4500, because the key turn falls outside the window in most conversations. Sending only the user turns cuts the input by 37% and holds the accuracy at 0.6750; when the model does not see its own earlier responses, it loses nothing. The extracted state record cuts the input by 66% and holds the accuracy at 0.6500.

The difference between the last two numbers is 0.0250. Both run columns need to be consulted: the full state row gives 0.7000 on the second seed instead of 0.8250, so the run difference of this scaffolding can reach 0.1250. 0.0250 is one fifth of that and counts as unmeasured; the 0.1750 between full state and the state record, on the other hand, exceeds the run difference and is measured.

The last column changes the ranking. The tokens per correct label are 1413.8 in the last three turns mode, and this is more expensive than the 1337.0 of the client that sends no state at all: a shortening decision can fall below both the full transcript and the client without scaffolding. The state record gives the cheapest correct result at 494.9. The price of the cheapness is the silence of the extractor: in five of the forty conversations no field is found and the client reports this not as an error but as yok (none). The last line states the limit: if the conversation continues, the full transcript exceeds the context window with 246 tokens at the twelfth turn and the call is never made. Window overflow was measured in the Introduction to AI Engineering course; the only thing added here is that the client’s own state decision drives it to that limit.

Summary

• Conversation state is not kept on the server; because the transcript is resent with every request, the total sent over an eight-turn conversation is 3.96 times the last turn.
• When no state is carried, accuracy stays at chance level with 0.2000; when the transcript is sent in full it rises to 0.8250 and input tokens grow 3.73 times.
• Sending the last three turns cuts the input by 26% but drops the accuracy to 0.4500; the key turn falls outside in most conversations.
• The extracted state record cuts the input by 66% and holds the accuracy at 0.6500; the 0.0250 gap with user turns only counts as unmeasured because the run difference of the two runs is 0.1250.
• Tokens per correct label change the ranking: the last three turns at 1413.8 are more expensive than the client without scaffolding, and the state record at 494.9 is the cheapest.
• The price of the cheap decision is silent loss: the extractor finds no field in five of forty conversations and does not report this as an error.

Next Step

In this lesson every response arrived as a single piece and the user waited for it to complete. The printed first token step entered no decision: while the growing prompt pushed the first token from 1 step to 5 steps, the expected time still stretched to the last chunk. Had the response reached our hands in the order it was produced instead of as a whole, which number changes and which stays the same. There is also the cut seen in the failure column: what is left when it happens in the middle of a response. The next lesson counts these three separately.