--- title: 'Who Is an AI Engineer' source: 'https://academia.sh/en/courses/introduction-to-ai-engineering/who-is-an-ai-engineer' course: 'Introduction to AI Engineering' language: en updated: '2026-10-21T05:02:30+00:00' license: 'CC BY-SA 4.0' objectives: - 'Builds a language model simulator and reads the measurable surface of a given component with five measures: input tokens, output tokens, context window, stop condition, and status.' - 'Compares the label hit rate of a bare call and a small scaffolding on the same task and computes the difference.' - 'Counts the cost of the scaffolding as tokens, calls, and failure share, and derives the unit cost per correct response.' --- # Who Is an AI Engineer Establishing the unit of measurement for building products without training a model: a language model simulator is written and the surface of a given component is read with five numbers. On the same 120 fictional support tickets, a bare call gives 0.5167 label hit rate and 0.3583 unlabeled response share, while a small scaffolding with a label set and a format constraint gives 0.7750 and 0.0000. The cost is written side by side: input tokens rise from 2666 to 6222, output tokens fall from 1648 to 218, the call count rises from 120 to 124, total tokens rise 1.49 times, and the cost per correct label stays near 69 tokens. Over 121 lessons, the previous curriculum built a model, measured it, explained it, and counted what it did to whom. Its closing left one gap written down: all of that happened **in a single run, on a single machine, inside a notebook**. A model turning into a **system** — getting versioned, gaining an interface that others call, being rolled back when it breaks, having its cost counted — was never covered. This curriculum opens from there and adds one more thing: the model is **no longer something you trained** either. It is a given component whose weights you cannot see, whose settings you cannot change, and whose insides you cannot inspect. The only thing left in your hands is its **surroundings**: what you say, what you put next to it, how you read its output, when you give up. What this curriculum measures is not the model but those surroundings; they will be called the **scaffolding**. This lesson sets up the unit of measurement and writes the **language model simulator** that will be used across seven courses. - **FR1.** No lesson in this curriculum calls a real model endpoint. The code calls a **simulator** written with the standard library. The simulator does not imitate a product; it carries the **measurable surface** of an endpoint. - **FR2.** The simulator's tokenizer is fictional: words are split on whitespace and pieces longer than five characters are cut. Tokenization itself was measured in the Natural Language Processing and Computer Vision course and is not repeated here; here the token is only a **unit of cost**. - **FR3.** The surface is five measures: input tokens, output tokens, the **context window** (240 tokens), maximum output length (24 tokens), and status. - **FR4.** The simulator is determined by a seed (**20260218**). The same prompt on the same seed gives the same response; the numbers do not drift from run to run. **Uncertainty is measured by changing the seed.** - **FR5.** Two failures are built in: the rate limit depends on the call count (every thirty-seventh call), and the timeout depends on prompt length. Both enter the failure share. - **FR6.** The support tickets are **fictional**: 120 records, five topics, and a misleading sentence appended to a portion of the records. The regional measurement network of the earlier curricula (Data Analytics and Machine Learning and Data Science) turns into a support assistant here. - **FR7.** The simulator's internal trace table holds a few **misleading links**. They are the fictional counterpart of the concept pairs a model systematically confuses, and they cannot be removed from the outside. - **FR8.** The criterion is the **label hit rate**. If a label cannot be read from the response, the record counts as wrong and is additionally shown in the **unlabeled response share**. ## The Measurable Surface of a Model A given component cannot be measured from the inside. Its weights are invisible, its training data is unknown, and the reasoning behind its decision cannot be read. One place to measure remains: the **surface the component presents to the outside**. How many tokens went in, how many came out, where the limit was, what the same input produced in two runs, which status the call returned with. **This is the rule of the course: a given component is measured from its surface.** The first block sets up this surface. What gets called is a language model simulator; no request goes to a real endpoint. ```python # SIMULATOR. This code does not call a real model endpoint; it is a language model # simulator written with the standard library. The support tickets are also FICTIONAL. import math SEED, M32 = 20260218, 0xFFFFFFFF WINDOW, MAX_OUTPUT = 240, 24 # context window and maximum output, in tokens def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(text): # FICTIONAL tokenizer: pieces of at most five characters tok = [] for s in text.lower().replace(",", " , ").split(): while len(s) > 5: tok.append(s[:5]) s = s[5:] tok.append(s) return tok def trace_id(text): # trace that depends on the prompt content, not on the run h = 2166136261 for c in text: h = ((h ^ ord(c)) * 16777619) & M32 return h TOPICS = ["invoice", "fault", "meter", "tariff", "outage"] TEMPLATES = { "invoice": ["the invoice amount this month is very high compared to last month", "i paid my invoice but the system shows a debt", "where can i see the invoice details"], "fault": ["the water is not flowing at home and there is a fault i think", "the connection is broken and it is not working at all", "i opened a fault ticket but the team never came"], "meter": ["the meter reading was entered wrong and the index does not match", "i want an appointment for the meter change", "the index on the meter and the invoice are not the same"], "tariff": ["i want to learn which tariff tier i am on", "what do i need to do for a tariff change", "does the unit price change by subscriber type"], "outage": ["there is an outage in the neighborhood when will it be back", "where is the planned outage announcement tracked", "we have been having outages since the evening"], } EXTRAS = ["the same issue happened last month too", "i am not satisfied with this", "i urgently need information", "i do not have my record number"] TICKETS = [] for i in range(120): r = rng(SEED + i) k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] m = TEMPLATES[k][int(r() * 3)] if r() < 0.34: m += " " + EXTRAS[int(r() * 4)] TICKETS.append({"id": 4001 + i, "text": m, "topic": k}) CONCEPTS = TOPICS + ["payment", "complaint", "information", "general"] TRACES = {"invoi": "invoice", "amoun": "invoice", "detai": "invoice", "fault": "fault", "broke": "fault", "worki": "fault", "flowi": "fault", "meter": "meter", "index": "meter", "readi": "meter", "tarif": "tariff", "unit": "tariff", "subsc": "tariff", "tier": "tariff", "outag": "outage", "annou": "outage", "neigh": "outage", "debt": "payment", "paid": "payment", "price": "payment", "satis": "complaint", "issue": "complaint", "urgen": "complaint", "infor": "information", "learn": "information", "where": "information"} TRAPS = {"chang": "tariff", "recor": "fault", "high": "tariff", "shows": "invoice", "team": "outage", "track": "tariff", "enter": "invoice"} FREE_FORM = ["your request seems to relate to {k}", "this falls under {k} , forwarded to the relevant team", "your record has been received , we will review the matter and get back to you"] COUNTER = [0] def call(prompt, seed=SEED, temperature=0.6, max_output=MAX_OUTPUT): COUNTER[0] += 1 tok = tokenize(prompt) if len(tok) > WINDOW: return {"status": "context_overflow", "input": len(tok), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: return {"status": "rate_limit", "input": len(tok), "output": 0, "response": ""} r = rng(seed ^ trace_id(prompt)) if len(tok) > 90 and r() < 0.04: return {"status": "timeout", "input": len(tok), "output": 0, "response": ""} cand = [k for k in CONCEPTS if k[:5] in tok] # concepts named in the prompt if len(cand) < 3: cand = CONCEPTS p = {k: 0.15 for k in cand} for t in tok: for table, w in ((TRACES, 1.0), (TRAPS, 1.3)): if table.get(t) in p: p[table[t]] += w if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) # most likely choice else: e = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [v / sum(e) for v in e])] y = k if "only" in tok else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k) out = tokenize(y) if len(out) > max_output: # stop condition out = out[:max_output] y = " ".join(out) return {"status": "ok", "input": len(tok), "output": len(out), "response": y} def read_label(y): # extract a label from the response for s in y.lower().split(): if s in TOPICS: return s return None first = TICKETS[0] pieces = tokenize(first["text"]) print(f"FICTIONAL ticket set: {len(TICKETS)} records, {len(TOPICS)} topics") print(f"example ticket {first['id']}: {first['text']}") print(f"token count {len(pieces)}, first six pieces {pieces[:6]}") sample = call("Answer this support ticket: " + first["text"]) print(f"\nstatus {sample['status']}, input {sample['input']} tokens, output {sample['output']} tokens") print(f"response: {sample['response']}") prompt1 = "Answer this support ticket: " + TICKETS[3]["text"] for sd in (SEED, SEED + 1): print(f"seed {sd}: {call(prompt1, seed=sd)['response']}") short = call("Answer this support ticket: " + first["text"], max_output=4) print(f"\nmaximum output 4 tokens: {short['response']}") long_call = call(" ".join(x["text"] for x in TICKETS[:20])) print(f"prompt over the window: status {long_call['status']}, input {long_call['input']} (limit {WINDOW})") ``` ``` FICTIONAL ticket set: 120 records, 5 topics example ticket 4001: where is the planned outage announcement tracked token count 12, first six pieces ['where', 'is', 'the', 'plann', 'ed', 'outag'] status ok, input 19 tokens, output 10 tokens response: your request seems to relate to outage seed 20260218: your request seems to relate to outage seed 20260219: this falls under fault , forwarded to the relevant team maximum output 4 tokens: your reque st seems prompt over the window: status context_overflow, input 321 (limit 240) ``` Each of the seven lines after the header shows a limit. A ticket of seven words comes to **twelve tokens**; a word longer than five characters is split into pieces. The number of words and the number of tokens are not the same thing, and the cost is read from the second. The fourth and fifth lines are the most uncomfortable feature of this curriculum. **The same prompt, on two seeds, gave two different responses**: the first names `outage`, a label that is not the ticket's topic, while the second says `fault`. The ticket's real topic is `fault`. So the correct response came out once and did not come out once, and in between the prompt never changed. A component giving two different responses to the same input is the justification for everything that will be built around it. The last two lines are two hard limits. When the maximum output length is lowered to four, the response is cut after its fourth token: **the stop condition cut the response in the middle** and the output stopped on a fragment, not a finished answer. When twenty tickets are joined and sent, 321 tokens exceed the context window and the call returns without producing any response. Neither is the model "thinking wrong"; both are numbers given by the surface. ## The Difference Between a Bare Call and Scaffolding The task is fixed: each of 120 fictional support tickets will be assigned to one of five topics. Two paths are put side by side. The **bare call** is the plainest use — sending the ticket as it is and reading the response. The **scaffolding** is three small parts: stating the label set in the prompt, imposing the format, and retrying once on an unreadable response. ```python INSTRUCTION = ("Assign the ticket to one of these labels: invoice , fault , meter , tariff , " "outage . Write only the label as the response . Ticket: ") def measure(scaffolding): COUNTER[0] = 0 tin = tout = hits = empty = retries = failures = 0 for x in TICKETS: prompt = (INSTRUCTION if scaffolding else "Answer this support ticket: ") + x["text"] s = call(prompt) tin, tout = tin + s["input"], tout + s["output"] failures += s["status"] != "ok" e = read_label(s["response"]) if e is None and scaffolding: # format could not be read: retry once retries += 1 s = call(prompt, seed=SEED + 1) tin, tout = tin + s["input"], tout + s["output"] failures += s["status"] != "ok" e = read_label(s["response"]) empty += e is None hits += e == x["topic"] return {"hit": hits / len(TICKETS), "empty": empty / len(TICKETS), "calls": COUNTER[0], "input": tin, "output": tout, "retries": retries, "failure": failures / COUNTER[0]} A, B = measure(False), measure(True) print(f"{'measure':<24} {'bare call':>13} {'scaffolding':>11} {'difference':>10}") for name, key, fmt in (("label hit rate", "hit", "{:.4f}"), ("unlabeled response share", "empty", "{:.4f}"), ("call count", "calls", "{:d}"), ("input tokens", "input", "{:d}"), ("output tokens", "output", "{:d}"), ("failure share", "failure", "{:.4f}")): f = B[key] - A[key] print(f"{name:<24} {fmt.format(A[key]):>13} {fmt.format(B[key]):>11} " f"{('{:+.4f}' if '.' in fmt else '{:+d}').format(f):>10}") tA, tB = A["input"] + A["output"], B["input"] + B["output"] print(f"\ntotal tokens {tA} -> {tB} ({tB / tA:.2f}x), retries {B['retries']}") print(f"cost per correct label: {tA / A['hit'] / len(TICKETS):.1f} -> " f"{tB / B['hit'] / len(TICKETS):.1f} tokens") ``` ``` measure bare call scaffolding difference label hit rate 0.5167 0.7750 +0.2583 unlabeled response share 0.3583 0.0000 -0.3583 call count 120 124 +4 input tokens 2666 6222 +3556 output tokens 1648 218 -1430 failure share 0.0250 0.0242 -0.0008 total tokens 4314 -> 6440 (1.49x), retries 4 cost per correct label: 69.6 -> 69.2 tokens ``` The first row is the scaffolding's gain: **from 0.5167 to 0.7750**, an increase of **0.2583**. The second row says where the gain comes from. In the bare call, **0.3583 of the responses** carried no readable label at all; either the response closed on a word outside the five labels, or, in the form of "your record has been received , we will review the matter and get back to you", it carried no topic name. When the label set is written into the prompt, this share drops to **zero**. The gain does not come from the model thinking better but from **the answer becoming readable**. ## A Scaffolding Without a Written Cost Counts as Unmeasured The lower half of the same table is the cost, and in this curriculum it is as binding as the upper half. Input tokens went **from 2666 to 6222**: adding the instruction to the start of every ticket more than doubled the prompt length. Output tokens went the other way, **from 1648 to 218**; producing a one-word response is several times cheaper than producing a sentence. The call count went **from 120 to 124**; all four extra calls are requests that hit the rate limit and were retried. The failure share stayed roughly the same on both paths, because the rate limit is a property of the **surface**, not of the scaffolding. The last row is the real number of this lesson. Total tokens rose **1.49 times**: the scaffolding spends half again as many tokens on the same 120 tickets. The hit rate rose by almost the same factor, so the cost per correct label barely moves: **from 69.6 tokens to 69.2 tokens**. The scaffolding did not pay for its accuracy with a cheaper label; it **bought more correct labels at a higher total bill**. A report that writes only the hit rate, or only the per-label cost, hides the 1.49 times; a report that writes only one of the two numbers is a defense, not a measurement. **This is the rule of the curriculum: the number of an artificial intelligence system is not the model's number but the scaffolding's number; a scaffolding without a written cost counts as unmeasured.** The work of the AI engineer is visible in this table. The model is given, and in this table it never changed: the same simulator, the same seed, the same weights. The only thing that changed is its **surroundings** — the label set written into the prompt, the imposed format, the retry decision made on an unreadable response, and the latency class held to a single call. The settings learned over 121 lessons are not in hand here; other settings take their place, and all of them sit **outside** the model. The next lesson puts these two sets of settings side by side and counts them. ## Summary - A given component cannot be measured from the inside; it is measured from its **surface**: input tokens, output tokens, context window, stop condition, and status. The code of this curriculum calls a **simulator**. - The model is an uncertain component. The same prompt gave two different responses on two seeds; in one the correct label came out, in the other a wrong label. - On the same 120 fictional tickets, the bare call gives a label hit rate of **0.5167**, and **0.3583** of the responses have no readable label. The scaffolding made of a label set, a format constraint, and a single retry gives **0.7750** and **0.0000**. - The cost is written side by side: input tokens go from 2666 to 6222, calls from 120 to 124, output tokens fall from 1648 to 218, and total tokens rise **1.49 times**. - The cost per correct label stays nearly flat, **from 69.6 to 69.2 tokens**, while the total bill rises 1.49 times. A scaffolding without a written cost counts as unmeasured. ## Next Step In this lesson the model never changed and all of the gain came from outside it. This is the exact opposite of the previous curriculum: there the gain came from inside the model and the outside was held fixed. Both approaches can solve the same problem, but what they hold in their hands is not the same. The next lesson counts it: on the same support ticket problem, how many settings the training approach has, how many settings the using approach has, how many of those settings really move the number, and where **who holds which decision** separates in the two approaches.