Who Is an AI Engineer

Over 121 lessons, the previous curriculum built a model, measured it, explained it, and counted what it did to whom. Its closing left one gap written down: all of that happened in a single run, on a single machine, inside a notebook. A model turning into a system — getting versioned, gaining an interface that others call, being rolled back when it breaks, having its cost counted — was never covered.

This curriculum opens from there and adds one more thing: the model is no longer something you trained either. It is a given component whose weights you cannot see, whose settings you cannot change, and whose insides you cannot inspect. The only thing left in your hands is its surroundings: what you say, what you put next to it, how you read its output, when you give up. What this curriculum measures is not the model but those surroundings; they will be called the scaffolding. This lesson sets up the unit of measurement and writes the language model simulator that will be used across seven courses.

• FR1. No lesson in this curriculum calls a real model endpoint. The code calls a simulator written with the standard library. The simulator does not imitate a product; it carries the measurable surface of an endpoint.
• FR2. The simulator’s tokenizer is fictional: words are split on whitespace and pieces longer than five characters are cut. Tokenization itself was measured in the Natural Language Processing and Computer Vision course and is not repeated here; here the token is only a unit of cost.
• FR3. The surface is five measures: input tokens, output tokens, the context window (240 tokens), maximum output length (24 tokens), and status.
• FR4. The simulator is determined by a seed (20260218). The same prompt on the same seed gives the same response; the numbers do not drift from run to run. Uncertainty is measured by changing the seed.
• FR5. Two failures are built in: the rate limit depends on the call count (every thirty-seventh call), and the timeout depends on prompt length. Both enter the failure share.
• FR6. The support tickets are fictional: 120 records, five topics, and a misleading sentence appended to a portion of the records. The regional measurement network of the earlier curricula (Data Analytics and Machine Learning and Data Science) turns into a support assistant here.
• FR7. The simulator’s internal trace table holds a few misleading links. They are the fictional counterpart of the concept pairs a model systematically confuses, and they cannot be removed from the outside.
• FR8. The criterion is the label hit rate. If a label cannot be read from the response, the record counts as wrong and is additionally shown in the unlabeled response share.

The Measurable Surface of a Model

A given component cannot be measured from the inside. Its weights are invisible, its training data is unknown, and the reasoning behind its decision cannot be read. One place to measure remains: the surface the component presents to the outside. How many tokens went in, how many came out, where the limit was, what the same input produced in two runs, which status the call returned with. This is the rule of the course: a given component is measured from its surface.

The first block sets up this surface. What gets called is a language model simulator; no request goes to a real endpoint.

# SIMULATOR. This code does not call a real model endpoint; it is a language model
# simulator written with the standard library. The support tickets are also FICTIONAL.
import math

SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW, MAX_OUTPUT = 240, 24        # context window and maximum output, in tokens


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(text):                 # FICTIONAL tokenizer: pieces of at most five characters
    tok = []
    for s in text.lower().replace(",", " , ").split():
        while len(s) > 5:
            tok.append(s[:5])
            s = s[5:]
        tok.append(s)
    return tok


def trace_id(text):                 # trace that depends on the prompt content, not on the run
    h = 2166136261
    for c in text:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["invoice", "fault", "meter", "tariff", "outage"]
TEMPLATES = {
    "invoice": ["the invoice amount this month is very high compared to last month",
                "i paid my invoice but the system shows a debt",
                "where can i see the invoice details"],
    "fault": ["the water is not flowing at home and there is a fault i think",
              "the connection is broken and it is not working at all",
              "i opened a fault ticket but the team never came"],
    "meter": ["the meter reading was entered wrong and the index does not match",
              "i want an appointment for the meter change",
              "the index on the meter and the invoice are not the same"],
    "tariff": ["i want to learn which tariff tier i am on",
               "what do i need to do for a tariff change",
               "does the unit price change by subscriber type"],
    "outage": ["there is an outage in the neighborhood when will it be back",
               "where is the planned outage announcement tracked",
               "we have been having outages since the evening"],
}
EXTRAS = ["the same issue happened last month too", "i am not satisfied with this",
          "i urgently need information", "i do not have my record number"]

TICKETS = []
for i in range(120):
    r = rng(SEED + i)
    k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    m = TEMPLATES[k][int(r() * 3)]
    if r() < 0.34:
        m += " " + EXTRAS[int(r() * 4)]
    TICKETS.append({"id": 4001 + i, "text": m, "topic": k})

CONCEPTS = TOPICS + ["payment", "complaint", "information", "general"]
TRACES = {"invoi": "invoice", "amoun": "invoice", "detai": "invoice",
          "fault": "fault", "broke": "fault", "worki": "fault", "flowi": "fault",
          "meter": "meter", "index": "meter", "readi": "meter",
          "tarif": "tariff", "unit": "tariff", "subsc": "tariff", "tier": "tariff",
          "outag": "outage", "annou": "outage", "neigh": "outage",
          "debt": "payment", "paid": "payment", "price": "payment",
          "satis": "complaint", "issue": "complaint", "urgen": "complaint",
          "infor": "information", "learn": "information", "where": "information"}
TRAPS = {"chang": "tariff", "recor": "fault", "high": "tariff", "shows": "invoice",
         "team": "outage", "track": "tariff", "enter": "invoice"}
FREE_FORM = ["your request seems to relate to {k}",
             "this falls under {k} , forwarded to the relevant team",
             "your record has been received , we will review the matter and get back to you"]
COUNTER = [0]


def call(prompt, seed=SEED, temperature=0.6, max_output=MAX_OUTPUT):
    COUNTER[0] += 1
    tok = tokenize(prompt)
    if len(tok) > WINDOW:
        return {"status": "context_overflow", "input": len(tok), "output": 0, "response": ""}
    if COUNTER[0] % 37 == 0:
        return {"status": "rate_limit", "input": len(tok), "output": 0, "response": ""}
    r = rng(seed ^ trace_id(prompt))
    if len(tok) > 90 and r() < 0.04:
        return {"status": "timeout", "input": len(tok), "output": 0, "response": ""}
    cand = [k for k in CONCEPTS if k[:5] in tok]    # concepts named in the prompt
    if len(cand) < 3:
        cand = CONCEPTS
    p = {k: 0.15 for k in cand}
    for t in tok:
        for table, w in ((TRACES, 1.0), (TRAPS, 1.3)):
            if table.get(t) in p:
                p[table[t]] += w
    if temperature <= 0.0:
        k = max(sorted(p), key=lambda a: p[a])       # most likely choice
    else:
        e = [math.exp(p[a] / temperature) for a in sorted(p)]
        k = sorted(p)[pick(r(), [v / sum(e) for v in e])]
    y = k if "only" in tok else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k)
    out = tokenize(y)
    if len(out) > max_output:                        # stop condition
        out = out[:max_output]
        y = " ".join(out)
    return {"status": "ok", "input": len(tok), "output": len(out), "response": y}


def read_label(y):                                   # extract a label from the response
    for s in y.lower().split():
        if s in TOPICS:
            return s
    return None


first = TICKETS[0]
pieces = tokenize(first["text"])
print(f"FICTIONAL ticket set: {len(TICKETS)} records, {len(TOPICS)} topics")
print(f"example ticket {first['id']}: {first['text']}")
print(f"token count {len(pieces)}, first six pieces {pieces[:6]}")
sample = call("Answer this support ticket: " + first["text"])
print(f"\nstatus {sample['status']}, input {sample['input']} tokens, output {sample['output']} tokens")
print(f"response: {sample['response']}")
prompt1 = "Answer this support ticket: " + TICKETS[3]["text"]
for sd in (SEED, SEED + 1):
    print(f"seed {sd}: {call(prompt1, seed=sd)['response']}")
short = call("Answer this support ticket: " + first["text"], max_output=4)
print(f"\nmaximum output 4 tokens: {short['response']}")
long_call = call(" ".join(x["text"] for x in TICKETS[:20]))
print(f"prompt over the window: status {long_call['status']}, input {long_call['input']} (limit {WINDOW})")

FICTIONAL ticket set: 120 records, 5 topics
example ticket 4001: where is the planned outage announcement tracked
token count 12, first six pieces ['where', 'is', 'the', 'plann', 'ed', 'outag']

status ok, input 19 tokens, output 10 tokens
response: your request seems to relate to outage
seed 20260218: your request seems to relate to outage
seed 20260219: this falls under fault , forwarded to the relevant team

maximum output 4 tokens: your reque st seems
prompt over the window: status context_overflow, input 321 (limit 240)

Each of the seven lines after the header shows a limit. A ticket of seven words comes to twelve tokens; a word longer than five characters is split into pieces. The number of words and the number of tokens are not the same thing, and the cost is read from the second.

The fourth and fifth lines are the most uncomfortable feature of this curriculum. The same prompt, on two seeds, gave two different responses: the first names outage, a label that is not the ticket’s topic, while the second says fault. The ticket’s real topic is fault. So the correct response came out once and did not come out once, and in between the prompt never changed. A component giving two different responses to the same input is the justification for everything that will be built around it.

The last two lines are two hard limits. When the maximum output length is lowered to four, the response is cut after its fourth token: the stop condition cut the response in the middle and the output stopped on a fragment, not a finished answer. When twenty tickets are joined and sent, 321 tokens exceed the context window and the call returns without producing any response. Neither is the model “thinking wrong”; both are numbers given by the surface.

The Difference Between a Bare Call and Scaffolding

The task is fixed: each of 120 fictional support tickets will be assigned to one of five topics. Two paths are put side by side. The bare call is the plainest use — sending the ticket as it is and reading the response. The scaffolding is three small parts: stating the label set in the prompt, imposing the format, and retrying once on an unreadable response.

INSTRUCTION = ("Assign the ticket to one of these labels: invoice , fault , meter , tariff , "
               "outage . Write only the label as the response . Ticket: ")


def measure(scaffolding):
    COUNTER[0] = 0
    tin = tout = hits = empty = retries = failures = 0
    for x in TICKETS:
        prompt = (INSTRUCTION if scaffolding else "Answer this support ticket: ") + x["text"]
        s = call(prompt)
        tin, tout = tin + s["input"], tout + s["output"]
        failures += s["status"] != "ok"
        e = read_label(s["response"])
        if e is None and scaffolding:        # format could not be read: retry once
            retries += 1
            s = call(prompt, seed=SEED + 1)
            tin, tout = tin + s["input"], tout + s["output"]
            failures += s["status"] != "ok"
            e = read_label(s["response"])
        empty += e is None
        hits += e == x["topic"]
    return {"hit": hits / len(TICKETS), "empty": empty / len(TICKETS), "calls": COUNTER[0],
            "input": tin, "output": tout, "retries": retries, "failure": failures / COUNTER[0]}


A, B = measure(False), measure(True)
print(f"{'measure':<24} {'bare call':>13} {'scaffolding':>11} {'difference':>10}")
for name, key, fmt in (("label hit rate", "hit", "{:.4f}"),
                       ("unlabeled response share", "empty", "{:.4f}"),
                       ("call count", "calls", "{:d}"), ("input tokens", "input", "{:d}"),
                       ("output tokens", "output", "{:d}"),
                       ("failure share", "failure", "{:.4f}")):
    f = B[key] - A[key]
    print(f"{name:<24} {fmt.format(A[key]):>13} {fmt.format(B[key]):>11} "
          f"{('{:+.4f}' if '.' in fmt else '{:+d}').format(f):>10}")
tA, tB = A["input"] + A["output"], B["input"] + B["output"]
print(f"\ntotal tokens {tA} -> {tB} ({tB / tA:.2f}x), retries {B['retries']}")
print(f"cost per correct label: {tA / A['hit'] / len(TICKETS):.1f} -> "
      f"{tB / B['hit'] / len(TICKETS):.1f} tokens")

measure                      bare call scaffolding difference
label hit rate                  0.5167      0.7750    +0.2583
unlabeled response share        0.3583      0.0000    -0.3583
call count                         120         124         +4
input tokens                      2666        6222      +3556
output tokens                     1648         218      -1430
failure share                   0.0250      0.0242    -0.0008

total tokens 4314 -> 6440 (1.49x), retries 4
cost per correct label: 69.6 -> 69.2 tokens

The first row is the scaffolding’s gain: from 0.5167 to 0.7750, an increase of 0.2583. The second row says where the gain comes from. In the bare call, 0.3583 of the responses carried no readable label at all; either the response closed on a word outside the five labels, or, in the form of “your record has been received , we will review the matter and get back to you”, it carried no topic name. When the label set is written into the prompt, this share drops to zero. The gain does not come from the model thinking better but from the answer becoming readable.

A Scaffolding Without a Written Cost Counts as Unmeasured

The lower half of the same table is the cost, and in this curriculum it is as binding as the upper half. Input tokens went from 2666 to 6222: adding the instruction to the start of every ticket more than doubled the prompt length. Output tokens went the other way, from 1648 to 218; producing a one-word response is several times cheaper than producing a sentence. The call count went from 120 to 124; all four extra calls are requests that hit the rate limit and were retried. The failure share stayed roughly the same on both paths, because the rate limit is a property of the surface, not of the scaffolding.

The last row is the real number of this lesson. Total tokens rose 1.49 times: the scaffolding spends half again as many tokens on the same 120 tickets. The hit rate rose by almost the same factor, so the cost per correct label barely moves: from 69.6 tokens to 69.2 tokens. The scaffolding did not pay for its accuracy with a cheaper label; it bought more correct labels at a higher total bill. A report that writes only the hit rate, or only the per-label cost, hides the 1.49 times; a report that writes only one of the two numbers is a defense, not a measurement. This is the rule of the curriculum: the number of an artificial intelligence system is not the model’s number but the scaffolding’s number; a scaffolding without a written cost counts as unmeasured.

The work of the AI engineer is visible in this table. The model is given, and in this table it never changed: the same simulator, the same seed, the same weights. The only thing that changed is its surroundings — the label set written into the prompt, the imposed format, the retry decision made on an unreadable response, and the latency class held to a single call. The settings learned over 121 lessons are not in hand here; other settings take their place, and all of them sit outside the model. The next lesson puts these two sets of settings side by side and counts them.

Summary

• A given component cannot be measured from the inside; it is measured from its surface: input tokens, output tokens, context window, stop condition, and status. The code of this curriculum calls a simulator.
• The model is an uncertain component. The same prompt gave two different responses on two seeds; in one the correct label came out, in the other a wrong label.
• On the same 120 fictional tickets, the bare call gives a label hit rate of 0.5167, and 0.3583 of the responses have no readable label. The scaffolding made of a label set, a format constraint, and a single retry gives 0.7750 and 0.0000.
• The cost is written side by side: input tokens go from 2666 to 6222, calls from 120 to 124, output tokens fall from 1648 to 218, and total tokens rise 1.49 times.
• The cost per correct label stays nearly flat, from 69.6 to 69.2 tokens, while the total bill rises 1.49 times. A scaffolding without a written cost counts as unmeasured.

Next Step

In this lesson the model never changed and all of the gain came from outside it. This is the exact opposite of the previous curriculum: there the gain came from inside the model and the outside was held fixed. Both approaches can solve the same problem, but what they hold in their hands is not the same. The next lesson counts it: on the same support ticket problem, how many settings the training approach has, how many settings the using approach has, how many of those settings really move the number, and where who holds which decision separates in the two approaches.