Skip to content
academia.sh
tr
Log in

Unit 01 of 04

API Usage

Measuring what a message-based interface offers and what each capability costs in tokens: carrying the conversation state on the client and the resending that grows as it lengthens, with the three shortening decisions that bring it down; the steps until the first token and how much of a response cut off midway stays usable; the valid call rate and a fabricated tool name caught by validation; the accuracy and token difference between prompt-level and interface-level enforcement; the tokens the same information costs as text, grid and sequence; the number stored and the neighborhood accuracy as the embedding dimension drops; the number of fields a compatible API layer carries and loses; the number of calls that falls and the number of affected requests that grows as batch size grows.

Published Updated

Lessons

  1. 01 Message-Based Interfaces The token cost of carrying conversation state on the client: in an eight-turn conversation the total input tokens sent is 3.96 times the last turn, because every turn resends the whole transcript. Across forty fictional conversations, five send policies are measured side by side: a client that sends no state gets 0.2000 accuracy with 8346 input tokens, the full transcript 0.8250 with 31160, the last three turns 0.4500 with 23102, the user turns only 0.6750 with 19738, an extracted state record 0.6500 with 10474. Reading tokens per correct label changes the ranking: the last three turns at 1413.8 cost more than a client that does nothing, and the extracted state record at 494.9 is the cheapest. The difference between the two runs is 0.1250, so the 0.0250 gap between the state record and user only counts as unmeasured. 13 min read
  2. 02 Response Streaming Measuring receiving the response chunk by chunk: on the same 320-call workload all three setups spend 23338 input and 4049 output tokens and are cut midway in 33 calls, that is, streaming does not change a single token. What changes is the waiting step: the time to first token is 16.03 steps on average in the non-streaming client and 3.29 steps in the streaming client. But streaming alone does not change usability; when the label is at the end of the response, the time to actionable output is 16.03 in both setups and the number recovered from the 33 cut responses is 0. When the label is moved to the start of the response, the time to actionable output falls to 4.96, the number recovered from the cut ones rises to 31 and the usable share rises from 0.8625 to 0.9594. On average 0.5801 of the cut responses has reached us, and the field order decides whether this share is of any use. 13 min read
  3. 03 Tool and Function Calling Measuring how a model output is tied to a call that will be executed: on 120 fictional requests, when the tool catalog is not written into the prompt the valid call rate is 0.0500 and 0.6750 of the responses fabricate a tool name that does not exist. When only the names are given, the fabricated name share falls to 0.1500 but the parameter error jumps from 0.0667 to 0.5583: the error does not disappear, it moves. When names and parameters are given together, the valid rate is 0.6917 and the rate of calling the correct tool is 0.5750, and the price is input growing from 3694 to 11614 tokens. Still, the tokens per correct tool fall from 750.5 to 178.2. Validation catches 37 calls, but of the 83 calls counted as valid, 14 call a registered but wrong tool, and no schema catches this residue. The format error does not shrink with the catalog, because its source is the network, not the prompt. 14 min read
  4. 04 Structured Output Mode Measuring side by side writing the schema into the prompt and giving it to the interface: on a four-field schema, writing it into the prompt spends 7654 input tokens and gives 0.7583 conformance, while the interface field gives 0.9000 with 4294 tokens. When the schema grows to twelve fields, conformance with the prompt falls to 0.6083, with the interface it stays fixed at 0.9000 and the saving rises to 6360 tokens. When the network layer is left out, the conformance of interface enforcement is exactly 1.0000, and this is the number measured by the Prompt Engineering course; when the network is counted, the same number falls to 0.9000, that is, the remaining error share of schema enforcement comes not from the model but from the network. The tokens per correct record fall from 125.8 to 65.7 at four fields and from 287.1 to 123.0 at twelve fields. Adding fields grows the output tokens in both modes but does not raise the correct record rate. 14 min read
  5. 05 Multimodal Input Measuring the same field inspection record in six input formats: the text summary 21, the cropped fine grid 42, the coarse grid 59, the fine grid 183 tokens, and the sequence between 195 and 330 per record. The usable record rate is 0.0833 on the coarse grid, 0.8167 on the fine grid and 0.7000 on the cropped grid; the cropped format delivers about eighty-six percent of the fine grid's usable records with a quarter of the tokens, and the cost per record falls from 224.1 to 60.0. Although the sequence format has the highest visibility, it stays at 0.3667 because it exceeds the context window in thirty-four of sixty records, and it becomes the most expensive format. The audio mode reads the index at 0.8833 but, being blind to the body condition, produces no usable record. 11 min read
  6. 06 Embedding Interfaces Measuring an endpoint that returns a vector instead of text in the response, and a single decision: how many dimensions to request for the embedding. Over seventy-two fictional records the cost of the call is independent of the dimension and stays fixed at 1814 input tokens; what changes is the stored numbers (from 10368 to 648) and the comparison multiplications (from 368064 to 23004). The share of the three nearest neighbors being from the same topic is 0.9259 at 144 dimensions, 0.9028 at 72, 0.8102 at 36, 0.6389 at 9. The loss from halving the dimension is 0.0231, the difference from a second embedding model at the same dimension is 0.0232: halving cuts the storage in half at an unmeasurable loss. As the dimension falls, the difference between the two models grows from 0.0232 to 0.1018, and the gain of folding on the client side does not exceed this difference. 13 min read
  7. 07 Compatible Interface Layers Counting, field by field, two fictional providers and the common interface between them: provider A offers twelve fields, eight are carried and four are lost; provider B offers ten fields, eight are carried and two are lost. Because the stop sequence field exists only in A, it finds no place in the intersection and the output tokens rise from 773 to 1638. Of the hundred and twenty calls, 117 are cut at the length limit; both providers report this in their own interface, and the number of cuts readable from the common interface is 0. With the same common call, temperature 0.6 gives 0.5500 accuracy on A and 0.5000 on B; because temperatures 0.4 and 0.6 are rounded to the same setting on B, both give 0.5000 there while A separates between 0.7583 and 0.5500. 13 min read
  8. 08 Batch and Asynchronous Calls Processing a hundred and twenty fictional requests at seven different batch sizes, and batch size moving three separate numbers in opposite directions: input tokens fall from 5854 to 1784, calls from 120 to 10, and rounds at four-way concurrency from 30 to 3. The load other than the repeated directive is exactly 1414 tokens in all seven rows; the whole saving comes from the repeated directive. In return, the number of requests affected by a single failure equals the batch size and rises from 1 to 12; at size 18 the prompt overflows the context window, six of seven calls drop and accuracy collapses to 0.0667. The movement of accuracy with batch size does not exceed two run differences, that is, it counts as unmeasured. If the output limit is not enlarged with the batch size, 26 requests go unanswered at size 12 and accuracy falls to 0.5333. 6 min read

Start typing to search.

↑↓ ↵ Esc navigate · open · close