Batch and Asynchronous Calls

Up to the previous lesson every measure was written under the assumption of one call per request: a hundred and twenty requests, a hundred and twenty calls. This assumption has an invisible cost. The same directive is placed at the head of every call and is paid for again each time. When a nightly batch job has thousands of records instead of a hundred and twenty, this repetition alone makes up most of the bill.

A batch call removes it: the directive is written once, numbered requests are lined up behind it and the response is read in one go. This lesson treats batch size as a setting and prints three numbers side by side: calls and tokens, waiting, and the number of requests affected by a single failure. The first two fall as batch size grows, the third grows.

• AU63. The code is a simulator; no real endpoint is called. The requests and all the failure rules are fictional, seed 20260218, second run 20260219.
• AU64. Failure is at call level. Whatever the reason a call is rejected, all the requests inside it are rejected. This is the definition of a batch interface and it is what this lesson actually measures.
• AU65. There are three sources of rejection: the context window (240 tokens), the rate limit (one call in thirty-seven) and malformed output. The last is a share of one and a half percent per request; if one request spoils it, the whole response drops.
• AU66. That the malformed-output share is per request is the key assumption of this lesson. The underlying defect rate is independent of batch size; the only thing that changes is how many requests one defect takes with it.
• AU67. Waiting is written in rounds. Four calls are sent concurrently; the number of rounds is the number of calls divided by four, rounded up. Raw time is not written; the round count does not change from run to run.
• AU68. By default the output limit grows with the batch size (three tokens per request and a margin of two shares). What happens when it is not enlarged is measured separately.
• AU69. The response is split into rows by a decoder: a number, then a label. If a row cannot be read, that request counts as unanswered, even if the call succeeded.
• AU70. The procedure for queues, backoff and asynchronous job management was measured in the Caching, Queues and Asynchronous Processing course of the Backend Development curriculum; it is not repeated. This lesson adds only what batch size does to the three numbers.

How Many Requests Fit in One Call

The prompt of a batch call is a single piece: the directive once, then the numbered requests. The response is also a single piece, and to be read it has to be split into rows. The block below builds these two ends and shows what happens at three batch sizes.

# SIMULATOR -- FICTIONAL. No real model endpoint is called. The requests and
# all the failure rules are fictional.
import math
SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW, CONCURRENCY = 240, 4            # context window , number of concurrent calls


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):                        # FICTIONAL tokenizer, at most five characters per piece
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def hash_text(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"]
PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"),
            ("faturami odedim ama sistemde borc gorunuyor", "fatura"),
            ("fatura ayrintisini nereden gorebilirim", "fatura"),
            ("evde su akmiyor bir ariza var galiba", "ariza"),
            ("baglanti bozuk oldu ve hic calismiyor", "ariza"),
            ("ariza kaydi actim ama ekip gelmedi", "ariza"),
            ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"),
            ("sayac degisimi icin randevu istiyorum", "sayac"),
            ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"),
            ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"),
            ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"),
            ("birim ucret abone tipine gore degisiyor mu", "tarife"),
            ("mahallede kesinti var ne zaman gelecek", "kesinti"),
            ("planli kesinti duyurusu nereden takip edilir", "kesinti"),
            ("aksamdan beri kesinti yasiyoruz", "kesinti")]
REQUESTS = []
for i in range(120):                    # 120 FICTIONAL support requests
    r = rng(SEED + i)
    k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    REQUESTS.append({"id": 4001 + i, "topic": k,
                     "text": [x for x in PATTERNS if x[1] == k][int(r() * 3)][0]})
CLUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza",
         "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac",
         "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife",
         "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti",
         "mahal": "kesinti"}
PREFIX = ("Her talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife ,"
          " kesinti . Her satir icin once numara sonra tek etiket yaz . Talepler: ")
PREFIX5 = {k[:5]: k for k in TOPICS}
CALL_COUNT = [0]


def batch_call(batch, seed=SEED, max_length=None, temperature=0.6):
    """Sends k requests in a single call. Failure is at call level:
    whichever rejection occurs, the whole batch is rejected."""
    CALL_COUNT[0] += 1
    k = len(batch)
    prompt = PREFIX + " ".join(f"{i + 1} {x['text']} ;" for i, x in enumerate(batch))
    toks = tokenize(prompt)
    max_length = 3 * k + 2 if max_length is None else max_length
    empty = {"input": len(toks), "output": 0, "response": []}
    if len(toks) > WINDOW:
        return dict(empty, status="context_overflow")
    if CALL_COUNT[0] % 37 == 0:
        return dict(empty, status="rate_limit")
    r = rng(seed ^ hash_text(prompt))
    for _ in batch:                      # each request carries its own malformed-output share
        if r() < 0.015:                 # if one request spoils it, the whole response drops
            return dict(empty, status="malformed")
    cb = []
    for i, x in enumerate(batch):
        p = {c: 0.15 for c in TOPICS}
        for t in tokenize(x["text"]):
            if CLUES.get(t) in p:
                p[CLUES[t]] += 1.0
        names = sorted(p)
        w = [math.exp(p[a] / temperature) for a in names]
        cb += [str(i + 1)] + tokenize(names[pick(r(), [q / sum(w) for q in w])])
    cut = len(cb) > max_length
    return {"status": "done", "input": len(toks), "output": min(len(cb), max_length),
            "response": cb[:max_length], "cut": cut}


def decode(cb, k):
    """Reads one label per row from the batch response. A row that cannot be read stays None."""
    out, i = [None] * k, 0
    while i < len(cb):
        if cb[i].isdigit():
            n, j = int(cb[i]), i + 1
            piece = []
            while j < len(cb) and not cb[j].isdigit():
                piece.append(cb[j])
                j += 1
            e = "".join(piece)
            if 1 <= n <= k and e[:5] in PREFIX5:
                out[n - 1] = PREFIX5[e[:5]]
            i = j
        else:
            i += 1
    return out


for k in (1, 6, 18):
    s = batch_call(REQUESTS[:k])
    print(f"batch size {k:>2} : input {s['input']:>3} , output {s['output']:>2} ,"
          f" status {s['status']}")
s = batch_call(REQUESTS[:6])
print("\nresponse of a single six-request call:", " ".join(s["response"]))
print("decoded:", decode(s["response"], 6))
print("actual :", [x["topic"] for x in REQUESTS[:6]])
d = batch_call(REQUESTS[:6], max_length=8)   # if the output limit is not enlarged with the batch size
print(f"\nsame call with max_length 8: output {d['output']} , cut {d['cut']} ,"
      f" decoded {sum(e is not None for e in decode(d['response'], 6))} / 6")

batch size  1 : input  50 , output  3 , status done
batch size  6 : input 107 , output 16 , status done
batch size 18 : input 247 , output  0 , status context_overflow

response of a single six-request call: 1 kesin ti 2 kesin ti 3 kesin ti 4 ariza 5 sayac 6 fatur a
decoded: ['kesinti', 'kesinti', 'kesinti', 'ariza', 'sayac', 'fatura']
actual : ['kesinti', 'fatura', 'ariza', 'ariza', 'sayac', 'tarife']

same call with max_length 8: output 8 , cut True , decoded 3 / 6

The first three lines show the ceiling of batch size. The directive is thirty-seven tokens; a call with a single request comes to fifty tokens, that is, three quarters of the input is directive. In the six-request call the input rises to 107, but the share per request falls to eighteen. At eighteen requests the prompt reaches 247 tokens and exceeds the 240-token window: the call is rejected without ever being made. The upper limit of batch size is set by the context window, not by the rate limit.

The decoder shows the second face of a batch response. The response is a single sequence of tokens, and the rows can be separated only by their numbers. All six rows are read, three correctly. The last block shows what happens when the output limit is not enlarged with the batch size: a limit of eight tokens cuts the response in the middle, the call returns successfully and only three of the six requests are decoded. A cut does not show in the call status; it becomes visible only if the rows the decoder leaves empty are counted.

The Three Numbers of Batch Size

A hundred and twenty requests are processed from start to finish at seven different batch sizes. Each row prints the number of calls, the input and output tokens, the failed calls, the number of requests affected by those failures, the delivered accuracy, the second run accuracy and the number of rounds at four-way concurrency.

def run(k, seed=SEED, max_length=None):
    CALL_COUNT[0] = 0
    calls = t_in = t_out = failed = affected = correct = unanswered = 0
    for i in range(0, 120, k):
        batch = REQUESTS[i:i + k]
        s = batch_call(batch, seed, max_length)
        calls, t_in, t_out = calls + 1, t_in + s["input"], t_out + s["output"]
        if s["status"] != "done":
            failed, affected = failed + 1, affected + len(batch)
            continue
        for x, e in zip(batch, decode(s["response"], len(batch))):
            correct, unanswered = correct + (e == x["topic"]), unanswered + (e is None)
    return calls, t_in, t_out, failed, affected, correct / 120, -(-calls // CONCURRENCY), unanswered


print(" size  calls  input output     failed   affected accuracy    second run rounds")
for k in (1, 2, 3, 6, 12, 18, 24):
    a, b = run(k), run(k, SEED + 1)
    print(f"{k:>5} {a[0]:>6} {a[1]:>6} {a[2]:>6} {a[3]:>10} {a[4]:>10}"
          f" {a[5]:>8.4f} {b[5]:>13.4f} {a[6]:>6}")
for k in (6, 12):
    a = run(k, max_length=24)
    print(f"output limit fixed at 24 , size {k:>2} : unanswered requests {a[7]:>2} ,"
          f" accuracy {a[5]:.4f}")

 size  calls  input output     failed   affected accuracy    second run rounds
    1    120   5854    300          3          3   0.8250        0.7083     30
    2     60   3634    300          3          6   0.7667        0.7250     15
    3     40   2894    291          3          9   0.7167        0.6667     10
    6     20   2154    299          1          6   0.7833        0.7583      5
   12     10   1784    284          1         12   0.7083        0.6833      3
   18      7   1673     30          6        108   0.0667        0.0750      2
   24      5   1599      0          5        120   0.0000        0.0000      2
output limit fixed at 24 , size  6 : unanswered requests  0 , accuracy 0.7833
output limit fixed at 24 , size 12 : unanswered requests 26 , accuracy 0.5333

The whole drop in the input column comes from a single item, and it can be separated out exactly. The directive is thirty-seven tokens; it is paid for 120 times when sending one by one and 10 times when sending in batches of twelve, that is, it falls from 4440 tokens to 370. The remaining load is exactly 1414 tokens in all seven rows: the requests themselves, their numbers and their separators. A batch call removes not the load of the request but the repetition of the directive. The output column confirms this reading: three tokens per request and about three hundred in total, independent of batch size. A batch call does not save on the output.

The rounds column is the measure of waiting. When a hundred and twenty separate calls are sent four at a time it takes thirty rounds; with batches of twelve, three rounds. The latency class moves from the single call to the batch call and the difference between them is a factor of ten. Raw time is not written, but the round count does not change from run to run and is enough for the comparison.

The Input That Wins, the Radius That Grows

The affected requests column is the real finding of this lesson and it reads in a single line: the number of affected requests is the number of failed calls times the batch size. When sending one by one, three failures take three requests. In batches of six a single failure takes six requests, in batches of twelve it takes twelve. The underlying defect rate did not change; every request carries the same share of one and a half percent. The only thing that changes is the radius of one defect.

That the number of failed calls falls as batch size grows can hide this reading. It goes from three to one and on its own sounds like good news. Yet part of this fall is only because the number of calls has decreased: the rate limit kicks in once in thirty-seven calls and never kicks in for a setup that makes ten calls. Doing the same work with fewer calls does genuinely dilute the rate limit, but it does not dilute the malformed output; its share is per request, and the number of requests is fixed.

The rows for eighteen and twenty-four are beyond a threshold. At eighteen the prompt does not fit the window; six of the seven calls drop without ever being made and 108 requests go unanswered. The one call left standing is the last twelve requests, and accuracy falls to 0.0667. At twenty-four no call is made at all, and accuracy is zero. The added mechanism making the work worse is in this table not a curve but a cliff: raising the batch size from twelve to eighteen gains only 111 tokens of input and in return takes the whole job away.

Reading the accuracy column calls for a yardstick. In the first five rows accuracy wanders between 0.7083 and 0.8250; the difference between them is 0.1167. The second run column has, in the same rows, a largest difference of 0.1167 (0.8250 and 0.7083 when sending one by one). The effect of batch size on accuracy does not exceed the run-to-run movement, that is, it counts as unmeasured. This is not surprising: the label decision is made per request and sending in a batch does not change that decision. Batch size is not a quality setting, it is a cost and risk setting. Defending it on quality grounds, or blaming it on quality grounds, goes against the table.

The last two rows measure the case where the output limit is forgotten. When the limit is held fixed at twenty-four, batches of six are not affected, because six rows fit into twenty tokens. In batches of twelve, however, the tail of the response is cut and 26 requests go unanswered; accuracy falls from 0.7083 to 0.5333. This loss does not show in the failed calls column: the calls returned successfully. Enlarging the batch size makes it mandatory to enlarge the output limit with it.

Summary

• A batch call removes the repetition of the directive: input tokens fall from 5854 to 1784, because the 37-token directive is paid for 10 times instead of 120. The remaining load is exactly 1414 tokens in all seven rows and the output tokens are independent of batch size.
• Waiting is written in rounds: at four-way concurrency it falls from 30 rounds to 3.
• The number of requests affected by a single failure equals the batch size; the same defect rate goes from taking 3 requests to taking 12.
• The context window sets the upper limit of batch size: at size 18 the prompt rises to 247 tokens, six of seven calls drop without ever being made and accuracy falls to 0.0667, at size 24 to 0.0000.
• The 0.1167 wobble of accuracy in the first five rows stays within two run differences: batch size is not a quality setting but a cost and risk setting.
• If the output limit is not enlarged with the batch size, 26 requests go unanswered at size 12 and accuracy falls to 0.5333; this loss never shows in the failed calls column.

Next Step

In eight lessons, what the message-based interface offers was measured one by one: the cost of conversation state, what response streaming changes, the validity rate of tool calling, interface-level schema enforcement, the token bill of the input format, the trade-off of embedding dimension, the fields lost by the common interface and the three numbers of batch size. Under all eight lay the same assumption: the call succeeds.

In this lesson it was counted that the assumption does not hold, but no answer was given. Rejected calls were not retried, the batch that exceeded the window was not split, the twenty-six unanswered requests were not rescued, the repeated directive was not cached. They were counted and left there. The next topic starts exactly here and in its first lesson builds the second claim of this course: retrying a request can rescue it, but it also multiplies the tokens and strains the rate limit further. Adding resilience is not free, and sometimes it spoils the very number it set out to fix.