Skip to content
academia.sh
tr
Log in

Unit 02 of 04

Production Quality

Measuring the cost and the failure points of the resilience scaffolding built against calls that are slow, rejected or cut off midway: fixed-interval retry compared with backoff retry under load, and retries multiplying tokens as well; the stale-response share that grows as the cache hit rate rises; the latency-class and accuracy trade-off of the small context, the large context and the two-stage setup; the mean of tokens per request against the share of the most expensive one percent in the total; the accuracy of requests that fall to the fallback model dropping silently and staying invisible when it is not measured; counting the items that stop the same input from giving different outputs.

Published Updated

Lessons

  1. 01 Rate Limits and Retries Running the same 160 fictional tickets under three policies at two load levels: the bare client completes 0.8562 at both loads and never hits the rate limit. At light load retry raises completion to 1.0000, at a cost of 25 extra calls and 1209 tokens. At heavy load the same scaffolding does damage: fixed-interval retry falls to 0.8000 and produces 151 rate limit rejections, while backoff retry stays 0.0188 above the bare client at 0.8750, but the run-to-run difference is also 0.0188. The interactive flow that shares the same limit drops from 0.9500 to 0.8667: retry rescues its own work while breaking someone else's. 12 min read
  2. 02 Prompt Caching Measuring the same 240 fictional requests under seven cache setups: 175 of the prompt's 186 tokens are shared by every request. The prefix cache, keyed by content, cuts input tokens from 44367 to 7472 with a 0.9792 hit rate and leaves the correct-response share unchanged. In the response cache, keyed by question, the hit rate rises from 0.4708 to 0.8792 as the lifetime grows and calls fall from 240 to 29, but the stale-response share climbs from 0.0208 to 0.4417 and the correct-response share falls from 0.9333 to 0.5542. Together the two caches cut input to 2946, fifteenfold, and the prefix hit rate falls from 0.9792 to 0.9359 once the response cache is on. 7 min read
  3. 03 Latency Optimization Comparing the same 240 fictional requests under five setups by latency class and accuracy: the small context holds the wait at 1.0000 steps in the single-call class but its accuracy is 0.7750, while the large context in the same class makes the user wait 3.0000 steps and gives 0.9292. The two-stage setup moves 0.2500 of the requests to the chain, cuts the wait to 1.7500 and input tokens to 19716, and loses only 0.0125 of accuracy. The parallel setup pulls the wait down to 1.5000 but at 47352 tokens costs more than the large context. Sending the second stage in batches raises the wait to 3.5542 without changing accuracy or tokens: a batch call is not a latency mechanism. 10 min read
  4. 04 Cost Monitoring Keeping the tokens-per-request ledger for a fictional workload of four hundred requests and counting the tail of the distribution: the mean is 81.6 tokens while the median is 69, the 99th percentile 224 and the largest request 303. The most expensive one percent spends 0.0299 of the total tokens, the most expensive ten percent 0.2124. Across two runs the mean goes from 81.6 to 83.0 while the share of the most expensive one percent goes from 0.0299 to 0.0409, that is, the stable one is the mean and the one that moves is the tail. When the bill is split into items, the instruction takes 0.3921, the conversation state 0.3658 and the ticket text, which carries the new information, only 0.1190. Three policies that shorten the conversation state bring tokens per request from 81.6 down to 50.5, but the difference narrows when read in tokens per correct answer. 14 min read
  5. 05 Error and Outage Management Measuring what dropping a request to a fallback tier does to quality when the primary endpoint does not respond: in the setup without fallback the outage turns 26 requests into errors, while in the setup with fallback errors fall to 2 and the total accuracy rises from 0.8450 to 0.8850, that is, the fallback looks like an improvement. Read against the no-outage baseline, the same number has fallen from 0.8975 to 0.8850, and the drop is 0.0125, below the run-to-run difference of 0.0200. When the ledger writes response provenance, the drop becomes visible: primary tier 0.9037, fallback tier 0.6667. When the outage share is raised from 0.00 to 0.35, the drop in total accuracy exceeds the run-to-run difference only at 0.18; the accuracy of the primary tier does not move across the four rows, what moves is the mix. 6 min read
  6. 06 Nondeterminism and Reproducibility Counting the items that determine the output of a model call and releasing each of them alone: 11 of the twelve items move the responses, and the largest movement is in the format word of the instruction (all responses change, accuracy falls from 0.9125 to 0.5525). Whitespace layout, which changes no meaning at all, moves 0.0700 of the responses, and client normalization moves 0.1300. The maximum output length, which alone moves nothing, drops accuracy from 0.5525 to 0.2675 when released together with the instruction word. Nine of the items are on the client and 3 on the provider, and when the three are released together 0.1150 of the responses move and accuracy drops by 0.0200: this is the threshold of the previous two lessons. 14 min read

Start typing to search.

↑↓ ↵ Esc navigate · open · close