Skip to content
academia.sh
tr
Log in

Course Intermediate

Building Applications with Model APIs

Published Updated

By the end of this course

Start course

01

API Usage

Measuring what a message-based interface offers and what each capability costs in tokens: carrying the conversation state on the client and the resending that grows as it lengthens, with the three shortening decisions that bring it down; the steps until the first token and how much of a response cut off midway stays usable; the valid call rate and a fabricated tool name caught by validation; the accuracy and token difference between prompt-level and interface-level enforcement; the tokens the same information costs as text, grid and sequence; the number stored and the neighborhood accuracy as the embedding dimension drops; the number of fields a compatible API layer carries and loses; the number of calls that falls and the number of affected requests that grows as batch size grows.

  1. 01 Message-Based Interfaces The token cost of carrying conversation state on the client: in an eight-turn conversation the total input tokens sent is 3.96 times the last turn, because every turn resends the whole transcript. Across forty fictional conversations, five send policies are measured side by side: a client that sends no state gets 0.2000 accuracy with 8346 input tokens, the full transcript 0.8250 with 31160, the last three turns 0.4500 with 23102, the user turns only 0.6750 with 19738, an extracted state record 0.6500 with 10474. Reading tokens per correct label changes the ranking: the last three turns at 1413.8 cost more than a client that does nothing, and the extracted state record at 494.9 is the cheapest. The difference between the two runs is 0.1250, so the 0.0250 gap between the state record and user only counts as unmeasured.
  2. 02 Response Streaming Measuring receiving the response chunk by chunk: on the same 320-call workload all three setups spend 23338 input and 4049 output tokens and are cut midway in 33 calls, that is, streaming does not change a single token. What changes is the waiting step: the time to first token is 16.03 steps on average in the non-streaming client and 3.29 steps in the streaming client. But streaming alone does not change usability; when the label is at the end of the response, the time to actionable output is 16.03 in both setups and the number recovered from the 33 cut responses is 0. When the label is moved to the start of the response, the time to actionable output falls to 4.96, the number recovered from the cut ones rises to 31 and the usable share rises from 0.8625 to 0.9594. On average 0.5801 of the cut responses has reached us, and the field order decides whether this share is of any use.
  3. 03 Tool and Function Calling Measuring how a model output is tied to a call that will be executed: on 120 fictional requests, when the tool catalog is not written into the prompt the valid call rate is 0.0500 and 0.6750 of the responses fabricate a tool name that does not exist. When only the names are given, the fabricated name share falls to 0.1500 but the parameter error jumps from 0.0667 to 0.5583: the error does not disappear, it moves. When names and parameters are given together, the valid rate is 0.6917 and the rate of calling the correct tool is 0.5750, and the price is input growing from 3694 to 11614 tokens. Still, the tokens per correct tool fall from 750.5 to 178.2. Validation catches 37 calls, but of the 83 calls counted as valid, 14 call a registered but wrong tool, and no schema catches this residue. The format error does not shrink with the catalog, because its source is the network, not the prompt.
  4. 04 Structured Output Mode Measuring side by side writing the schema into the prompt and giving it to the interface: on a four-field schema, writing it into the prompt spends 7654 input tokens and gives 0.7583 conformance, while the interface field gives 0.9000 with 4294 tokens. When the schema grows to twelve fields, conformance with the prompt falls to 0.6083, with the interface it stays fixed at 0.9000 and the saving rises to 6360 tokens. When the network layer is left out, the conformance of interface enforcement is exactly 1.0000, and this is the number measured by the Prompt Engineering course; when the network is counted, the same number falls to 0.9000, that is, the remaining error share of schema enforcement comes not from the model but from the network. The tokens per correct record fall from 125.8 to 65.7 at four fields and from 287.1 to 123.0 at twelve fields. Adding fields grows the output tokens in both modes but does not raise the correct record rate.
  5. 05 Multimodal Input Measuring the same field inspection record in six input formats: the text summary 21, the cropped fine grid 42, the coarse grid 59, the fine grid 183 tokens, and the sequence between 195 and 330 per record. The usable record rate is 0.0833 on the coarse grid, 0.8167 on the fine grid and 0.7000 on the cropped grid; the cropped format delivers about eighty-six percent of the fine grid's usable records with a quarter of the tokens, and the cost per record falls from 224.1 to 60.0. Although the sequence format has the highest visibility, it stays at 0.3667 because it exceeds the context window in thirty-four of sixty records, and it becomes the most expensive format. The audio mode reads the index at 0.8833 but, being blind to the body condition, produces no usable record.
  6. 06 Embedding Interfaces Measuring an endpoint that returns a vector instead of text in the response, and a single decision: how many dimensions to request for the embedding. Over seventy-two fictional records the cost of the call is independent of the dimension and stays fixed at 1814 input tokens; what changes is the stored numbers (from 10368 to 648) and the comparison multiplications (from 368064 to 23004). The share of the three nearest neighbors being from the same topic is 0.9259 at 144 dimensions, 0.9028 at 72, 0.8102 at 36, 0.6389 at 9. The loss from halving the dimension is 0.0231, the difference from a second embedding model at the same dimension is 0.0232: halving cuts the storage in half at an unmeasurable loss. As the dimension falls, the difference between the two models grows from 0.0232 to 0.1018, and the gain of folding on the client side does not exceed this difference.
  7. 07 Compatible Interface Layers Counting, field by field, two fictional providers and the common interface between them: provider A offers twelve fields, eight are carried and four are lost; provider B offers ten fields, eight are carried and two are lost. Because the stop sequence field exists only in A, it finds no place in the intersection and the output tokens rise from 773 to 1638. Of the hundred and twenty calls, 117 are cut at the length limit; both providers report this in their own interface, and the number of cuts readable from the common interface is 0. With the same common call, temperature 0.6 gives 0.5500 accuracy on A and 0.5000 on B; because temperatures 0.4 and 0.6 are rounded to the same setting on B, both give 0.5000 there while A separates between 0.7583 and 0.5500.
  8. 08 Batch and Asynchronous Calls Processing a hundred and twenty fictional requests at seven different batch sizes, and batch size moving three separate numbers in opposite directions: input tokens fall from 5854 to 1784, calls from 120 to 10, and rounds at four-way concurrency from 30 to 3. The load other than the repeated directive is exactly 1414 tokens in all seven rows; the whole saving comes from the repeated directive. In return, the number of requests affected by a single failure equals the batch size and rises from 1 to 12; at size 18 the prompt overflows the context window, six of seven calls drop and accuracy collapses to 0.0667. The movement of accuracy with batch size does not exceed two run differences, that is, it counts as unmeasured. If the output limit is not enlarged with the batch size, 26 requests go unanswered at size 12 and accuracy falls to 0.5333.

02

Production Quality

Measuring the cost and the failure points of the resilience scaffolding built against calls that are slow, rejected or cut off midway: fixed-interval retry compared with backoff retry under load, and retries multiplying tokens as well; the stale-response share that grows as the cache hit rate rises; the latency-class and accuracy trade-off of the small context, the large context and the two-stage setup; the mean of tokens per request against the share of the most expensive one percent in the total; the accuracy of requests that fall to the fallback model dropping silently and staying invisible when it is not measured; counting the items that stop the same input from giving different outputs.

  1. 01 Rate Limits and Retries Running the same 160 fictional tickets under three policies at two load levels: the bare client completes 0.8562 at both loads and never hits the rate limit. At light load retry raises completion to 1.0000, at a cost of 25 extra calls and 1209 tokens. At heavy load the same scaffolding does damage: fixed-interval retry falls to 0.8000 and produces 151 rate limit rejections, while backoff retry stays 0.0188 above the bare client at 0.8750, but the run-to-run difference is also 0.0188. The interactive flow that shares the same limit drops from 0.9500 to 0.8667: retry rescues its own work while breaking someone else's.
  2. 02 Prompt Caching Measuring the same 240 fictional requests under seven cache setups: 175 of the prompt's 186 tokens are shared by every request. The prefix cache, keyed by content, cuts input tokens from 44367 to 7472 with a 0.9792 hit rate and leaves the correct-response share unchanged. In the response cache, keyed by question, the hit rate rises from 0.4708 to 0.8792 as the lifetime grows and calls fall from 240 to 29, but the stale-response share climbs from 0.0208 to 0.4417 and the correct-response share falls from 0.9333 to 0.5542. Together the two caches cut input to 2946, fifteenfold, and the prefix hit rate falls from 0.9792 to 0.9359 once the response cache is on.
  3. 03 Latency Optimization Comparing the same 240 fictional requests under five setups by latency class and accuracy: the small context holds the wait at 1.0000 steps in the single-call class but its accuracy is 0.7750, while the large context in the same class makes the user wait 3.0000 steps and gives 0.9292. The two-stage setup moves 0.2500 of the requests to the chain, cuts the wait to 1.7500 and input tokens to 19716, and loses only 0.0125 of accuracy. The parallel setup pulls the wait down to 1.5000 but at 47352 tokens costs more than the large context. Sending the second stage in batches raises the wait to 3.5542 without changing accuracy or tokens: a batch call is not a latency mechanism.
  4. 04 Cost Monitoring Keeping the tokens-per-request ledger for a fictional workload of four hundred requests and counting the tail of the distribution: the mean is 81.6 tokens while the median is 69, the 99th percentile 224 and the largest request 303. The most expensive one percent spends 0.0299 of the total tokens, the most expensive ten percent 0.2124. Across two runs the mean goes from 81.6 to 83.0 while the share of the most expensive one percent goes from 0.0299 to 0.0409, that is, the stable one is the mean and the one that moves is the tail. When the bill is split into items, the instruction takes 0.3921, the conversation state 0.3658 and the ticket text, which carries the new information, only 0.1190. Three policies that shorten the conversation state bring tokens per request from 81.6 down to 50.5, but the difference narrows when read in tokens per correct answer.
  5. 05 Error and Outage Management Measuring what dropping a request to a fallback tier does to quality when the primary endpoint does not respond: in the setup without fallback the outage turns 26 requests into errors, while in the setup with fallback errors fall to 2 and the total accuracy rises from 0.8450 to 0.8850, that is, the fallback looks like an improvement. Read against the no-outage baseline, the same number has fallen from 0.8975 to 0.8850, and the drop is 0.0125, below the run-to-run difference of 0.0200. When the ledger writes response provenance, the drop becomes visible: primary tier 0.9037, fallback tier 0.6667. When the outage share is raised from 0.00 to 0.35, the drop in total accuracy exceeds the run-to-run difference only at 0.18; the accuracy of the primary tier does not move across the four rows, what moves is the mix.
  6. 06 Nondeterminism and Reproducibility Counting the items that determine the output of a model call and releasing each of them alone: 11 of the twelve items move the responses, and the largest movement is in the format word of the instruction (all responses change, accuracy falls from 0.9125 to 0.5525). Whitespace layout, which changes no meaning at all, moves 0.0700 of the responses, and client normalization moves 0.1300. The maximum output length, which alone moves nothing, drops accuracy from 0.5525 to 0.2675 when released together with the instruction word. Nine of the items are on the client and 3 on the provider, and when the three are released together 0.1150 of the responses move and accuracy drops by 0.0200: this is the threshold of the previous two lessons.

03

AI in the Development Workflow

Counting from the defensive side what a code assistant gains on a fictional codebase and what it misses: acceptance rate by task class across completion, explanation, and transformation uses, and the point where a bare call collapses without scaffolding; tokens and hit rate as the number of files grows, and the point where irrelevant context lowers the hit rate; how often a multi-step handoff must stop for an error to be caught before it spreads; the defect each of the acceptance check suite, the pattern scan, and the fabricated dependency existence check catches, and the residual none of them catches; the rate at which generated tests catch planted defects and the share of vacuous tests; the number of fields that fall into a secret class in text that enters the context, how many a filter removes, and the share of wrong among accepted suggestions.

  1. 01 Code Assistants Measuring completion, explanation, and transformation uses on a fictional codebase by acceptance rate: the call without context leads only seven of twenty-four tasks to acceptance, and the collapse is in completion (1/8) and transformation (2/8); the only class left standing is explanation (4/8). Open-file context gives 10/24, but the gain is narrow because the run-to-run difference of the same setup is 2/24. The whole codebase raises it to 18/24 at a cost of 1104 tokens rising to 4224 tokens; acceptance per thousand tokens falls from 6.34 to 4.26. In the run-independent expected acceptance column, transformation, even with the whole codebase (0.4675), stays below explanation with no context at all (0.6450).
  2. 02 Providing Context Measuring enlarging the context against selecting the context side by side on the same fictional codebase: sending all the files raises expected acceptance to 0.6921 and spends 4224 tokens and 120 files, while sending only the required files raises it to 0.8233 and spends 2274 tokens and 45 files. Every irrelevant file takes 0.0400 from expected acceptance and adds 624 tokens; there is no threshold, it starts with the first file. The token cost of the context provider is independent of its hit rate and is fixed at 2274, but expected acceptance varies between 0.4960 and 0.8233; for the provider to beat sending the whole codebase, it has to find the required files in about three of four tasks.
  3. 03 Agentic Coding Measuring the checkpoint interval when a multi-step job is handed off: when the 480 steps of sixty fictional jobs are run at six different intervals, stopping at every step gives 478/480 correct steps but asks for 479 stops, stopping every eight steps drops to 60 stops and falls to 355/480. Wasted steps per caught error rise from 1.00 to 5.64; missed errors from 1 to 21. The number of calls is almost constant across the six intervals (between 549 and 598), so the cost of stopping is not calls but human attention. Tokens per correct step are lowest at the three-step interval in both runs (146.8 and 144.9), so the most accurate interval and the cheapest interval are not the same.
  4. 04 Reviewing Generated Code Passing twenty-four generated patches on a fictional codebase through three defensive checks: the acceptance check suite fails five of the six behavioral defect classes, the pattern scan eight of eight findings, and the dependency existence check nine of nine fabricated names. Twenty-six of twenty-eight defects are caught, two slip through as format errors, and one of the missed ones enters the codebase without receiving any warning.
  5. 05 Documentation and Test Generation Measuring the discriminating power of a generated test suite: of fourteen generated tests run on the six mutated versions of the fictional codebase, two fail on clean code, nine of the remaining twelve give the same result on all seven versions, and only three distinguish a defect. Detection is 3/6, the ceiling reachable with the whole pool is 6/6, and a single hand-added alignment item makes it 4/6. In the generated documentation ten claims can be checked by machine, six are right, and one describes a function that does not exist.
  6. 06 Limits and Risks Counting the three risks of AI-assisted development: of the 456 context lines in twenty-four request bundles, 168 fall into a secret class; a filter that looks at field names removes all of them but cannot reach the 12 traces left in free text, so its coverage stays at 0.9333; nine of the generated pieces of code carry a dependency name that does not exist; and ten of the nineteen accepted suggestions are defective, so the share of wrong among the accepted is 0.5263, above the baseline share of 0.4583.

04

Local Models and Adaptation

Counting the tradeoffs of running a model on your own hardware and of adapting it: the number of values that parameter count and context length hold in memory and the concurrent requests a budget affords, the memory that falls as the number of digits drops and the point where accuracy falls and which task it falls on first, the prompt-side mechanism and fine-tuning measured side by side on the same task and the case where fine-tuning does not win, the updated-parameter share of full update, layer freezing and low-rank update with its accuracy and the count of stored values, the gap a small model trained on a large model's output closes and the gap it leaves open, and the deployment options that four classes of usage conditions close in one scenario.

  1. 01 Self-Hosted Models Running a model on your own hardware makes the unit memory instead of tokens: the weight memory of three fictional scales is 70.5 million, 367.5 million and 2.14 billion values, the intermediate state per request in a 960-token context is 11.8 million, 47.2 million and 157.3 million values, the concurrent requests a three-billion-value budget affords are 248, 55 and 5, and the label accuracy on the same 120 fictional tickets is 0.6333, 0.8167 and 0.8667. The product of concurrent requests and accuracy falls from 157.1 to 4.3, so the largest scale yields one thirty-sixth of the correct-response capacity. With the distribution temperature at zero both runs give the same 0.8667, and at 0.02 28 of the 120 tickets change from run to run.
  2. 02 Quantization Measuring on the same 120 fictional tickets how the number of digits a weight is stored with affects memory and accuracy: at 16 digits weight memory is 34.3 billion digits and concurrent requests are 5, at 8 digits 17.2 billion and 24, at 4 digits 8.6 billion and 62. Five-class topic assignment is 0.8667 at 16 digits, stays at 0.8750 at 8 digits, and falls to 0.8000 at 4 digits, 0.6333 at 3 digits and 0.2083 at 2 digits; the two-class urgency distinction at the last four levels gives 0.9417, 0.9333, 0.8583 and 0.3583. Because the mean decision margin is 0.1597 in five classes and 0.2665 in two classes, the fine distinction goes first. The distinguishing difference of the 121 rows is erased completely in 24 rows at 4 digits and in 53 rows at 3 digits.
  3. 03 Fine-Tuning Measuring side by side on the same 60 fictional test records what happens when the same labeled examples are put into the prompt and when they are written into the weights: the baseline without scaffolding is 0.6000, fine-tuning on four examples moves no weight and stays at 0.6000, at eight examples it reaches 0.7333 and passes the 0.6500 of prompt scaffolding, at sixteen examples 0.8833. At thirty-two examples the two paths overlap in the band 0.9000 to 0.9500, at sixty examples prompt scaffolding gives 0.9667 and fine-tuning stays at 0.9500: fine-tuning does not win. The cost is in two separate units; prompt scaffolding raises the context per request from 28.48 to 623.48 tokens, that is 21.9 times, while fine-tuning is a one-time 180 update steps and moves a 0.2992 share of the 605 weights.
  4. 04 Adaptation Methods Measuring side by side, in the same field adaptation, the updated-parameter share and accuracy of full update, layer freezing and a low-rank update: on field spelling a fictional local model of 2292 numbers gives 0.2500 and 0.0833, full update 1.0000 and 0.9861, layer freezing that updates only the head 0.6944 and 0.7500, a low-rank update that updates 0.2094 of the parameters 0.9583 and 0.9306. Raising the rank from 3 to 4 makes 210 more numbers stored per adaptation and its mean gain is 0.0070, below the two-run gap. For adaptation to eight domains the stored count is 6132 against 18336, and as the hidden width grows the share of the low-rank update falls from 0.2094 to 0.0284.
  5. 05 Distillation Measuring, on two tasks, what a small model trained on 240 fictional records labeled by the large model gains over the same model trained on only six human labels: on the topic task it rises from 0.5000 to 0.6333 and captures the whole teacher accuracy, but it closes only a third of the gap to the 0.9000 given by 240 human labels. On the separate topic-count task distillation moves backward, falling from 0.5833 to 0.3667 and 0.2417, because the teacher does not count. Of the distilled model's errors, 0.9545 are on records where the teacher also errs; for the model trained on six labels this share is 0.3000.
  6. 06 Model Repositories and Terms of Use Counting how many distribution options four license condition classes close in a use scenario: of sixteen options built from four asset types and four channels, the usage limit alone closes 6, the redistribution right 3, the derivative-work condition 2 and the commercial-use record 2, a total of 13; together they close 10, because three options are closed by more than one condition. Four of the six options left open are in a single channel. Removing the usage limit opens three options, not six. When the monthly request volume triples, the open options fall from six to four, and at ten times they fall to one. This lesson is the close of the Building Applications with Model APIs course.

Final exam

Building Applications with Model APIs — Final Exam

In preparation

This course's question bank is not complete yet. The exam opens here automatically once enough questions are added.

Start typing to search.

↑↓ ↵ Esc navigate · open · close