--- title: 'Batch and Asynchronous Calls' source: 'https://academia.sh/en/courses/model-apis/batch-and-asynchronous-calls' course: 'Building Applications with Model APIs' language: en updated: '2026-10-21T05:02:32+00:00' license: 'CC BY-SA 4.0' objectives: - 'Writes with the standard library a batch interface that sends several requests in one call and a decoder that splits its response into rows.' - 'Measures the effect of batch size on the number of calls, the input tokens and the waiting rounds.' - 'Counts that the number of requests affected by a single failure equals the batch size and where the context window collapses.' - 'Compares the movement in accuracy with two run differences and shows that the output limit has to be enlarged with the batch size.' --- # Batch and Asynchronous Calls Processing a hundred and twenty fictional requests at seven different batch sizes, and batch size moving three separate numbers in opposite directions: input tokens fall from 5854 to 1784, calls from 120 to 10, and rounds at four-way concurrency from 30 to 3. The load other than the repeated directive is exactly 1414 tokens in all seven rows; the whole saving comes from the repeated directive. In return, the number of requests affected by a single failure equals the batch size and rises from 1 to 12; at size 18 the prompt overflows the context window, six of seven calls drop and accuracy collapses to 0.0667. The movement of accuracy with batch size does not exceed two run differences, that is, it counts as unmeasured. If the output limit is not enlarged with the batch size, 26 requests go unanswered at size 12 and accuracy falls to 0.5333. Up to the previous lesson every measure was written under the assumption of **one call per request**: a hundred and twenty requests, a hundred and twenty calls. This assumption has an invisible cost. The same directive is placed at the head of every call and is paid for again each time. When a nightly batch job has thousands of records instead of a hundred and twenty, this repetition alone makes up most of the bill. A batch call removes it: the directive is written once, numbered requests are lined up behind it and the response is read in one go. This lesson treats batch size as a setting and prints three numbers side by side: **calls and tokens**, **waiting**, and **the number of requests affected by a single failure**. The first two fall as batch size grows, the third grows. - **AU63.** The code is a **simulator**; no real endpoint is called. The requests and all the failure rules are **fictional**, seed **20260218**, second run **20260219**. - **AU64. Failure is at call level.** Whatever the reason a call is rejected, **all** the requests inside it are rejected. This is the definition of a batch interface and it is what this lesson actually measures. - **AU65.** There are three sources of rejection: the **context window** (240 tokens), the **rate limit** (one call in thirty-seven) and **malformed output**. The last is a share of one and a half percent **per request**; if one request spoils it, the whole response drops. - **AU66.** That the malformed-output share is **per request** is the key assumption of this lesson. The underlying defect rate is independent of batch size; the only thing that changes is **how many requests one defect takes with it**. - **AU67. Waiting is written in rounds.** Four calls are sent concurrently; the number of rounds is the number of calls divided by four, rounded up. **Raw time is not written**; the round count does not change from run to run. - **AU68.** By default the output limit grows with the batch size (three tokens per request and a margin of two shares). What happens when it is not enlarged is measured separately. - **AU69.** The response is split into rows by a **decoder**: a number, then a label. If a row cannot be read, that request counts as **unanswered**, even if the call succeeded. - **AU70.** The procedure for queues, backoff and asynchronous job management was measured in the Caching, Queues and Asynchronous Processing course of the Backend Development curriculum; it is **not repeated**. This lesson adds only what **batch size** does to the three numbers. ## How Many Requests Fit in One Call The prompt of a batch call is a single piece: the directive once, then the numbered requests. The response is also a single piece, and to be read it has to be split into rows. The block below builds these two ends and shows what happens at three batch sizes. ```python # SIMULATOR -- FICTIONAL. No real model endpoint is called. The requests and # all the failure rules are fictional. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, CONCURRENCY = 240, 4 # context window , number of concurrent calls def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, at most five characters per piece b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def hash_text(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["fatura", "ariza", "sayac", "tarife", "kesinti"] PATTERNS = [("bu ayki fatura tutari gecen aya gore cok yuksek geldi", "fatura"), ("faturami odedim ama sistemde borc gorunuyor", "fatura"), ("fatura ayrintisini nereden gorebilirim", "fatura"), ("evde su akmiyor bir ariza var galiba", "ariza"), ("baglanti bozuk oldu ve hic calismiyor", "ariza"), ("ariza kaydi actim ama ekip gelmedi", "ariza"), ("sayac okumasi yanlis girilmis endeks tutmuyor", "sayac"), ("sayac degisimi icin randevu istiyorum", "sayac"), ("sayac uzerindeki endeks ile fatura ayni degil", "sayac"), ("hangi tarife basamagindayim ogrenmek istiyorum", "tarife"), ("tarife degisikligi icin ne yapmam gerekiyor", "tarife"), ("birim ucret abone tipine gore degisiyor mu", "tarife"), ("mahallede kesinti var ne zaman gelecek", "kesinti"), ("planli kesinti duyurusu nereden takip edilir", "kesinti"), ("aksamdan beri kesinti yasiyoruz", "kesinti")] REQUESTS = [] for i in range(120): # 120 FICTIONAL support requests r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] REQUESTS.append({"id": 4001 + i, "topic": k, "text": [x for x in PATTERNS if x[1] == k][int(r() * 3)][0]}) CLUES = {"fatur": "fatura", "tutar": "fatura", "ayrin": "fatura", "ariza": "ariza", "bozuk": "ariza", "calis": "ariza", "akmiy": "ariza", "sayac": "sayac", "endek": "sayac", "okuma": "sayac", "tarif": "tarife", "birim": "tarife", "abone": "tarife", "basam": "tarife", "kesin": "kesinti", "duyur": "kesinti", "mahal": "kesinti"} PREFIX = ("Her talebi su etiketlerden birine ata: fatura , ariza , sayac , tarife ," " kesinti . Her satir icin once numara sonra tek etiket yaz . Talepler: ") PREFIX5 = {k[:5]: k for k in TOPICS} CALL_COUNT = [0] def batch_call(batch, seed=SEED, max_length=None, temperature=0.6): """Sends k requests in a single call. Failure is at call level: whichever rejection occurs, the whole batch is rejected.""" CALL_COUNT[0] += 1 k = len(batch) prompt = PREFIX + " ".join(f"{i + 1} {x['text']} ;" for i, x in enumerate(batch)) toks = tokenize(prompt) max_length = 3 * k + 2 if max_length is None else max_length empty = {"input": len(toks), "output": 0, "response": []} if len(toks) > WINDOW: return dict(empty, status="context_overflow") if CALL_COUNT[0] % 37 == 0: return dict(empty, status="rate_limit") r = rng(seed ^ hash_text(prompt)) for _ in batch: # each request carries its own malformed-output share if r() < 0.015: # if one request spoils it, the whole response drops return dict(empty, status="malformed") cb = [] for i, x in enumerate(batch): p = {c: 0.15 for c in TOPICS} for t in tokenize(x["text"]): if CLUES.get(t) in p: p[CLUES[t]] += 1.0 names = sorted(p) w = [math.exp(p[a] / temperature) for a in names] cb += [str(i + 1)] + tokenize(names[pick(r(), [q / sum(w) for q in w])]) cut = len(cb) > max_length return {"status": "done", "input": len(toks), "output": min(len(cb), max_length), "response": cb[:max_length], "cut": cut} def decode(cb, k): """Reads one label per row from the batch response. A row that cannot be read stays None.""" out, i = [None] * k, 0 while i < len(cb): if cb[i].isdigit(): n, j = int(cb[i]), i + 1 piece = [] while j < len(cb) and not cb[j].isdigit(): piece.append(cb[j]) j += 1 e = "".join(piece) if 1 <= n <= k and e[:5] in PREFIX5: out[n - 1] = PREFIX5[e[:5]] i = j else: i += 1 return out for k in (1, 6, 18): s = batch_call(REQUESTS[:k]) print(f"batch size {k:>2} : input {s['input']:>3} , output {s['output']:>2} ," f" status {s['status']}") s = batch_call(REQUESTS[:6]) print("\nresponse of a single six-request call:", " ".join(s["response"])) print("decoded:", decode(s["response"], 6)) print("actual :", [x["topic"] for x in REQUESTS[:6]]) d = batch_call(REQUESTS[:6], max_length=8) # if the output limit is not enlarged with the batch size print(f"\nsame call with max_length 8: output {d['output']} , cut {d['cut']} ," f" decoded {sum(e is not None for e in decode(d['response'], 6))} / 6") ``` ``` batch size 1 : input 50 , output 3 , status done batch size 6 : input 107 , output 16 , status done batch size 18 : input 247 , output 0 , status context_overflow response of a single six-request call: 1 kesin ti 2 kesin ti 3 kesin ti 4 ariza 5 sayac 6 fatur a decoded: ['kesinti', 'kesinti', 'kesinti', 'ariza', 'sayac', 'fatura'] actual : ['kesinti', 'fatura', 'ariza', 'ariza', 'sayac', 'tarife'] same call with max_length 8: output 8 , cut True , decoded 3 / 6 ``` The first three lines show the ceiling of batch size. The directive is thirty-seven tokens; a call with a single request comes to fifty tokens, that is, three quarters of the input is directive. In the six-request call the input rises to 107, but the share per request falls to eighteen. At eighteen requests the prompt reaches 247 tokens and exceeds the 240-token window: the call is rejected without ever being made. **The upper limit of batch size is set by the context window, not by the rate limit.** The decoder shows the second face of a batch response. The response is a single sequence of tokens, and the rows can be separated only by their numbers. All six rows are read, three correctly. The last block shows what happens when the output limit is not enlarged with the batch size: a limit of eight tokens cuts the response in the middle, the call **returns successfully** and only three of the six requests are decoded. A cut does not show in the call status; it becomes visible only if the rows the decoder leaves empty are counted. ## The Three Numbers of Batch Size A hundred and twenty requests are processed from start to finish at seven different batch sizes. Each row prints the number of calls, the input and output tokens, the failed calls, the **number of requests affected** by those failures, the delivered accuracy, the second run accuracy and the number of rounds at four-way concurrency. ```python def run(k, seed=SEED, max_length=None): CALL_COUNT[0] = 0 calls = t_in = t_out = failed = affected = correct = unanswered = 0 for i in range(0, 120, k): batch = REQUESTS[i:i + k] s = batch_call(batch, seed, max_length) calls, t_in, t_out = calls + 1, t_in + s["input"], t_out + s["output"] if s["status"] != "done": failed, affected = failed + 1, affected + len(batch) continue for x, e in zip(batch, decode(s["response"], len(batch))): correct, unanswered = correct + (e == x["topic"]), unanswered + (e is None) return calls, t_in, t_out, failed, affected, correct / 120, -(-calls // CONCURRENCY), unanswered print(" size calls input output failed affected accuracy second run rounds") for k in (1, 2, 3, 6, 12, 18, 24): a, b = run(k), run(k, SEED + 1) print(f"{k:>5} {a[0]:>6} {a[1]:>6} {a[2]:>6} {a[3]:>10} {a[4]:>10}" f" {a[5]:>8.4f} {b[5]:>13.4f} {a[6]:>6}") for k in (6, 12): a = run(k, max_length=24) print(f"output limit fixed at 24 , size {k:>2} : unanswered requests {a[7]:>2} ," f" accuracy {a[5]:.4f}") ``` ``` size calls input output failed affected accuracy second run rounds 1 120 5854 300 3 3 0.8250 0.7083 30 2 60 3634 300 3 6 0.7667 0.7250 15 3 40 2894 291 3 9 0.7167 0.6667 10 6 20 2154 299 1 6 0.7833 0.7583 5 12 10 1784 284 1 12 0.7083 0.6833 3 18 7 1673 30 6 108 0.0667 0.0750 2 24 5 1599 0 5 120 0.0000 0.0000 2 output limit fixed at 24 , size 6 : unanswered requests 0 , accuracy 0.7833 output limit fixed at 24 , size 12 : unanswered requests 26 , accuracy 0.5333 ``` The whole drop in the input column comes from a single item, and it can be separated out exactly. The directive is thirty-seven tokens; it is paid for 120 times when sending one by one and 10 times when sending in batches of twelve, that is, it falls from 4440 tokens to 370. The remaining load is **exactly 1414 tokens in all seven rows**: the requests themselves, their numbers and their separators. A batch call removes not the load of the request but **the repetition of the directive**. The output column confirms this reading: three tokens per request and about three hundred in total, independent of batch size. A batch call does not save on the output. The rounds column is the measure of waiting. When a hundred and twenty separate calls are sent four at a time it takes thirty rounds; with batches of twelve, three rounds. The latency class moves **from the single call to the batch call** and the difference between them is a factor of ten. Raw time is not written, but the round count does not change from run to run and is enough for the comparison. ## The Input That Wins, the Radius That Grows The affected requests column is the real finding of this lesson and it reads in a single line: **the number of affected requests is the number of failed calls times the batch size.** When sending one by one, three failures take three requests. In batches of six a single failure takes six requests, in batches of twelve it takes twelve. The underlying defect rate did not change; every request carries the same share of one and a half percent. The only thing that changes is **the radius of one defect**. That the number of failed calls falls as batch size grows can hide this reading. It goes from three to one and on its own sounds like good news. Yet part of this fall is only because **the number of calls has decreased**: the rate limit kicks in once in thirty-seven calls and never kicks in for a setup that makes ten calls. Doing the same work with fewer calls does genuinely dilute the rate limit, but it does not dilute the malformed output; its share is per request, and the number of requests is fixed. The rows for eighteen and twenty-four are beyond a threshold. At eighteen the prompt does not fit the window; six of the seven calls drop without ever being made and 108 requests go unanswered. The one call left standing is the last twelve requests, and accuracy falls to 0.0667. At twenty-four no call is made at all, and accuracy is zero. **The added mechanism making the work worse** is in this table not a curve but a cliff: raising the batch size from twelve to eighteen gains only 111 tokens of input and in return takes the whole job away. Reading the accuracy column calls for a yardstick. In the first five rows accuracy wanders between 0.7083 and 0.8250; the difference between them is 0.1167. The second run column has, in the same rows, a largest difference of 0.1167 (0.8250 and 0.7083 when sending one by one). **The effect of batch size on accuracy does not exceed the run-to-run movement**, that is, it counts as unmeasured. This is not surprising: the label decision is made per request and sending in a batch does not change that decision. Batch size is not a quality setting, it is a **cost and risk** setting. Defending it on quality grounds, or blaming it on quality grounds, goes against the table. The last two rows measure the case where the output limit is forgotten. When the limit is held fixed at twenty-four, batches of six are not affected, because six rows fit into twenty tokens. In batches of twelve, however, the tail of the response is cut and **26 requests go unanswered**; accuracy falls from 0.7083 to 0.5333. This loss does not show in the failed calls column: the calls returned successfully. Enlarging the batch size makes it mandatory to enlarge the output limit with it. ## Summary - A batch call removes the repetition of the directive: input tokens fall from 5854 to 1784, because the 37-token directive is paid for 10 times instead of 120. The remaining load is exactly 1414 tokens in all seven rows and the output tokens are independent of batch size. - Waiting is written in rounds: at four-way concurrency it falls from 30 rounds to 3. - The number of requests affected by a single failure equals the batch size; the same defect rate goes from taking 3 requests to taking 12. - The context window sets the upper limit of batch size: at size 18 the prompt rises to 247 tokens, six of seven calls drop without ever being made and accuracy falls to 0.0667, at size 24 to 0.0000. - The 0.1167 wobble of accuracy in the first five rows stays within two run differences: batch size is not a quality setting but a cost and risk setting. - If the output limit is not enlarged with the batch size, 26 requests go unanswered at size 12 and accuracy falls to 0.5333; this loss never shows in the failed calls column. ## Next Step In eight lessons, what the message-based interface offers was measured one by one: the cost of conversation state, what response streaming changes, the validity rate of tool calling, interface-level schema enforcement, the token bill of the input format, the trade-off of embedding dimension, the fields lost by the common interface and the three numbers of batch size. Under all eight lay the same assumption: **the call succeeds.** In this lesson it was counted that the assumption does not hold, but no answer was given. Rejected calls were not retried, the batch that exceeded the window was not split, the twenty-six unanswered requests were not rescued, the repeated directive was not cached. They were counted and left there. The next topic starts exactly here and in its first lesson builds the second claim of this course: retrying a request can rescue it, but it also multiplies the tokens and strains the rate limit further. Adding resilience is not free, and sometimes it spoils the very number it set out to fix.