Skip to content
academia.sh
tr
Log in

Course Beginner

Introduction to AI Engineering

Published Updated

By the end of this course

Start course

01

Field and Roles

Measuring with numbers what it means to build a product without training a model and to design around uncertain output: the difference and cost of a bare call versus a small scaffolding on the same support ticket task, the number of settings and the boundary of responsibility in the hands of a training approach and a using approach, in how many requests the output changes between two runs of the same input and how much of that the user sees, how much offering options, imposing a format, and grounding narrow the uncertainty, the separation of a measurable capability claim from an unmeasurable general claim, and how many of them can be tied to a test.

  1. 01 Who Is an AI Engineer Establishing the unit of measurement for building products without training a model: a language model simulator is written and the surface of a given component is read with five numbers. On the same 120 fictional support tickets, a bare call gives 0.5167 label hit rate and 0.3583 unlabeled response share, while a small scaffolding with a label set and a format constraint gives 0.7750 and 0.0000. The cost is written side by side: input tokens rise from 2666 to 6222, output tokens fall from 1648 to 218, the call count rises from 120 to 124, total tokens rise 1.49 times, and the cost per correct label stays near 69 tokens.
  2. 02 The Difference Between AI Engineering and Machine Learning Engineering Measuring the boundary of responsibility by the number of settings: on the same support ticket problem, two of the training approach's four settings and three of the using approach's four settings (four of five rows once the seed is counted) move the number by more than 0.0200. The training approach asks for 48 labeled examples and reaches 1.0000, while the using approach gives 0.7778 with zero labeled examples. No item of the two setting lists is shared. Retrying moves the number by 0.0139 and adds two calls and 111 tokens, while the seed, which is out of one's hands, moves the number by 0.0417.
  3. 03 Impact on Product Development Measuring with numbers what it means to design a product around uncertain output: when the same 72 fictional support tickets are called in two runs, on the plain prompt the response text changes in 0.8750 of the records while the decision underneath changes in 0.5972, and in 0.5556 of the records the correct response comes out in only one run. Three design decisions that narrow the uncertainty are measured side by side: imposing a format lowers the visible change from 0.8611 to 0.3056, grounding lowers it to 0.0972 and raises the hit rate to 0.8264 while adding 144 retrieval steps, and offering options raises the change to 0.4306 and doubles the calls.
  4. 04 The Distinction Between AI and General AI Separating the sentences made about a system by how testable they are: of fourteen fictional claims, five are measurable capability claims and nine are unmeasurable general claims, and one of these carries a number and a direction yet cannot be tested because it has no set. The share that can be tied to a test is 0.3571. Five tests are run on the simulator and four pass: the label hit rate comes out at 0.8264 against a 0.90 threshold and fails, the decision change share passes at 0.0972, the unlabeled response share passes at 0.0139, the document retrieval share passes at 0.9028, and the tokens per request pass at 75.2.

02

How Language Models Work

Measuring the surface a given language model exposes to the outside: the number of tokens the same Turkish text costs and how its agglutinative structure enlarges it, the cost of attention being able to use distant context as a number of comparisons, how many examples it takes for a task set to improve in pretraining and in fine-tuning, what the most likely choice and sampled choice give in two runs on the same prompt, what dropping from the head, dropping from the middle and summarizing gain in tokens and lose in hit rate when the context window is exceeded, the collapse of the bare call on counting and exact arithmetic tasks, the fabrication rate of claims not based on a document in an ungrounded response and the cost of three mitigations, the comparison of open- and closed-weight access on three metrics, how model selection changes as the metric weight changes, and the token and hit rate comparison of four scaffoldings.

  1. 01 Tokens Measuring the input unit of text for the model as a unit of cost: 300 fictional support tickets come to 4,674 words and 8,334 tokens, 1.7831 tokens per word; with suffix pieces removed 6,176 tokens remain and the suffix share comes out at 0.2589. Six roots take 6 tokens in plain form and 30 tokens as plural and possessive suffixes pile on. A bare call gives 27.78 tokens per ticket and a field hit rate of 0.9722, while the scaffolding that drops frequent pieces keeps the same hit rate with 11.68 tokens at the 0.50 threshold and falls to 6.23 tokens at 0.30, but the hit rate drops to 0.8611.
  2. 02 Transformer Intuition Measuring why attention can use distant context and what it costs as a number of comparisons: in 240 fictional ticket streams the topic object is on average 63.9 tokens from the end, and the reader that looks at the last 16 tokens gives a hit rate of 0.0000, returning the wrong object in 83 streams and nothing in 157. The last 64 tokens rise to 0.4917 but stay at 0.0000 beyond 65 tokens; the weight that decays with distance stays at 0.3333 although it sees every position. The weight independent of distance gives 1.0000 and its cost is 70.9 comparisons: in a full pass, when the context doubles the comparisons quadruple, and at 1024 tokens it is 8.26 times the fixed window.
  3. 03 Pretraining and Fine-Tuning Counting a task set split in two and how many examples each half needs to improve: the simulator pretrained on a fictional general-text corpus of 1,600 sentences, without seeing a single domain example, gives a next-token hit rate of 0.3186 on general text and 0.1983 on domain text; the same measure without pretraining is 0.1296. Fine-tuning with four hundred domain examples raises domain text to 0.5437 but drops general text to 0.2692. The organization's own fault-code mapping is never solved by pretraining: the hit rate stays at 0.0900, below the 0.1200 share of the most frequent code, and the 0.90 threshold is passed only with 400 labeled examples.
  4. 04 Inference The cost of producing a response step by step and a count for the selection rule: in 200 fictional tickets greedy decoding takes 6.43 tokens and 211.1 comparisons per response with a hit rate of 1.0000. When the maximum output length is pulled down to 4, all responses hit the limit and the hit rate drops to 0.0000, to 0.5450 at 6, to 0.8100 at 7. On the same prompt, as the distribution temperature of sampling rises from 0.2 to 2.0 the hit rate falls from 0.9900 to 0.5600 and the share of responses that change between two runs rises from 0.5550 to 0.8600; with greedy decoding this share is 0.0000. With the input fixed at 30 tokens, when the output grows to 512 tokens the output itself accounts for 0.8953 of the comparisons.
  5. 05 Context Window Measuring what happens when the window is exceeded and how many tokens three truncation decisions save and how much they lower the hit rate: in a window of 256 tokens, 0.5967 of 300 fictional prompts do not fit, and the bare call gives a full-response rate of 0.3833 at 282.3 tokens. Dropping from the head gives 0.7433 at 217.5 tokens, dropping from the middle gives 0.7367 at the same tokens, and summarizing raises the code hit rate to 0.9667 at 100.6 tokens but drops the step hit rate to 0.0000. Which decision truncation makes depends on the position of the evidence: dropping from the head gives 0.5351 in the head bucket, dropping from the middle gives 0.3627 in the middle bucket. When the window goes from 128 to 384, the full response rises from 0.3767 to 0.9867.
  6. 06 Capabilities and Limits Measuring five task classes with the bare call on twenty-four fictional support requests: summarization 0.7500, classification 0.8750, ranking 0.6250, counting 0.4167 and exact arithmetic 0.2917. When the seed is changed the number of the first three classes moves, the last two do not move at all, and the number of requests whose response changed comes out 0/24. The counting hit rate equals exactly the 10/24 requests that carry no inflected form, the exact-arithmetic hit rate equals exactly the 7/24 requests that carry no carry across a piece boundary. The three-run majority scaffolding changes nothing in these two classes with three times the tokens, the procedure scaffolding that takes the calculation out of the tokens carries both to 1.0000.
  7. 07 Hallucination When twenty-four questions are asked over a fictional document pool of twenty-seven facts, the ungrounded response gets a hit rate of 0.2917 and a fabrication rate of 0.6667, and the response changes in 16/24 requests across two runs. Three mitigations are measured side by side: grounding raises the hit rate by +0.3750 and brings fabrication down to 0.3333, claim verification brings fabrication down to 0.0857 but its cost is two calls and 66.2 tokens, and the abstention option drops the hit rate 0.0417 below the baseline and leaves 10/24 requests unanswered. All of the unsupported claims left after grounding come from questions that have no counterpart in the document, and the verification scaffolding gives a difference of 0/24 across two runs.
  8. 08 Closed- and Open-Weight Models Two access modes are compared on three criteria with numbers, and the criteria do not converge on a single form. The control held is 3/9 settings in the closed-weight form and 9/9 in the open-weight form. On unit cost the closed form holds a hit rate of 0.7917 at 41.8 tokens per request, while the open form reaches the same hit rate only after six examples are added, at 179.8 tokens, and the ratio comes out at 4.30. On the data boundary the closed form sends 24/24 requests, 1004 tokens and 72 numeric fields across the organization boundary, and the open form sends none. Two criteria pick the open-weight form, one picks the closed-weight form, and the three numbers have no common unit.
  9. 09 Model Selection Three candidates are built at the simulator level and counted on four criteria: small capability 0.5000 hit rate / 89.5 equivalent tokens / 7.29 steps / 0 requests crossing, medium capability 0.5833 / 195.3 / 7.00 / 0, large capability 0.7917 / 670.0 / 14.00 / 24. Across five weight sets the winner is medium three times, large once and small once. The weight sweep gives the thresholds: when the hit-rate weight reaches 4 the choice moves from medium to large, when the cost weight reaches 2 it moves from medium to small, and the latency and data-boundary weights do not change the choice up to twelve. When the same computation is repeated with seed 20260219, the winner of the hit-rate weighted set turns from large back to medium.
  10. 10 Token-Based Cost Over a fictional stream of ninety-six requests, four scaffoldings are compared in tokens and hit rate: the long prompt gives 0.7083 hit rate at 159.8 tokens per request, the short prompt with examples gives the same hit rate at 103.8 tokens, summarize and send gets down to 74.0 tokens with 0.8021, the hard summary drops the hit rate to 0.0833 at 68.0 tokens, and the cache gives 0.7500 at 17.7 tokens. The total tokens fall from 15,341 to 1,702 and the model calls fall from 96 to 23. When the budget drops to 7,000 tokens only two of the five scaffoldings fit, and the design is not chosen but imposed. The hit rate of the cache comes out at 0.7500 and 0.9583 across the two runs.

Final exam

Introduction to AI Engineering — Final Exam

In preparation

This course's question bank is not complete yet. The exam opens here automatically once enough questions are added.

Start typing to search.

↑↓ ↵ Esc navigate · open · close