The Difference Between AI Engineering and Machine Learning Engineering

The previous lesson raised the number from 0.5167 to 0.7750 without changing the model at all. All of the gain came from outside the model. In the previous curriculum the opposite held: the gain came from inside the model and the outside was held fixed. Both approaches can solve the same support ticket problem, but what they hold in their hands is not the same.

This lesson ties that difference to a number. Both approaches are run on the same 72-record test set, and the setting in each one’s hands is turned one by one. What is measured is not which is better but who holds which decision: how many settings there are, how many really move the number, and how much what is out of one’s hands moves it.

• FR9. The fiction and the simulator carry over unchanged from the previous lesson: 120 fictional support tickets, five topics, seed 20260218. The training set is the first 48 records and the test set is the remaining 72 records. Both approaches are read on the same test set.
• FR10. The training approach’s model is a count table: it keeps how many times each token in the training set appears under which topic and names the topic with the highest total evidence. It is a learning procedure and its weights are determined by training.
• FR11. The Supervised Learning course measured tuning variables (depth, minimum leaf, class weight, tuning budget). That procedure is not repeated here; here only who holds the setting is counted.
• FR12. The using approach calls the simulator and never touches the inside of the model. It builds no count table and sees no labeled examples.
• FR13. A setting “moves the number” when the absolute value of the hit rate difference between its two ends is greater than 0.0200. The threshold is a decision and is held fixed.
• FR14. The seed is not a setting; it is the measure of the simulator’s uncertainty. It is placed in the table to show how much what is out of one’s hands moves.
• FR15. The cost columns exist only for the using approach, because the cost of the training approach is not calls and tokens but labeled examples. The two cost units are not converted into each other.

The Settings in the Training Approach’s Hands

In the training approach the model is an object you produce. The settings in your hands look inside it: which data it is built from, which unit it counts with, which evidence it ignores, how it smooths sparse observations. The block below sets up the simulator and the fiction, then writes the learning procedure and turns four settings at two ends.

# SIMULATOR. This code does not call a real model endpoint; it is a language model
# simulator written with the standard library. The support tickets are also FICTIONAL.
import math

SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW, MAX_OUTPUT = 240, 24


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(text):
    tok = []
    for s in text.lower().replace(",", " , ").split():
        while len(s) > 5:
            tok.append(s[:5])
            s = s[5:]
        tok.append(s)
    return tok


def trace_id(text):
    h = 2166136261
    for c in text:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["invoice", "fault", "meter", "tariff", "outage"]
TEMPLATES = {
    "invoice": ["the invoice amount this month is very high compared to last month",
                "i paid my invoice but the system shows a debt",
                "where can i see the invoice details"],
    "fault": ["the water is not flowing at home and there is a fault i think",
              "the connection is broken and it is not working at all",
              "i opened a fault ticket but the team never came"],
    "meter": ["the meter reading was entered wrong and the index does not match",
              "i want an appointment for the meter change",
              "the index on the meter and the invoice are not the same"],
    "tariff": ["i want to learn which tariff tier i am on",
               "what do i need to do for a tariff change",
               "does the unit price change by subscriber type"],
    "outage": ["there is an outage in the neighborhood when will it be back",
               "where is the planned outage announcement tracked",
               "we have been having outages since the evening"],
}
EXTRAS = ["the same issue happened last month too", "i am not satisfied with this",
          "i urgently need information", "i do not have my record number"]

TICKETS = []
for i in range(120):
    r = rng(SEED + i)
    k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    m = TEMPLATES[k][int(r() * 3)]
    if r() < 0.34:
        m += " " + EXTRAS[int(r() * 4)]
    TICKETS.append({"id": 4001 + i, "text": m, "topic": k})

CONCEPTS = TOPICS + ["payment", "complaint", "information", "general"]
TRACES = {"invoi": "invoice", "amoun": "invoice", "detai": "invoice",
          "fault": "fault", "broke": "fault", "worki": "fault", "flowi": "fault",
          "meter": "meter", "index": "meter", "readi": "meter",
          "tarif": "tariff", "unit": "tariff", "subsc": "tariff", "tier": "tariff",
          "outag": "outage", "annou": "outage", "neigh": "outage",
          "debt": "payment", "paid": "payment", "price": "payment",
          "satis": "complaint", "issue": "complaint", "urgen": "complaint",
          "infor": "information", "learn": "information", "where": "information"}
TRAPS = {"chang": "tariff", "recor": "fault", "high": "tariff", "shows": "invoice",
         "team": "outage", "track": "tariff", "enter": "invoice"}
FREE_FORM = ["your request seems to relate to {k}",
             "this falls under {k} , forwarded to the relevant team",
             "your record has been received , we will review the matter and get back to you"]
COUNTER = [0]


def call(prompt, seed=SEED, temperature=0.6, max_output=MAX_OUTPUT):
    COUNTER[0] += 1
    tok = tokenize(prompt)
    if len(tok) > WINDOW:
        return {"status": "context_overflow", "input": len(tok), "output": 0, "response": ""}
    if COUNTER[0] % 37 == 0:
        return {"status": "rate_limit", "input": len(tok), "output": 0, "response": ""}
    r = rng(seed ^ trace_id(prompt))
    if len(tok) > 90 and r() < 0.04:
        return {"status": "timeout", "input": len(tok), "output": 0, "response": ""}
    cand = [k for k in CONCEPTS if k[:5] in tok]
    if len(cand) < 3:
        cand = CONCEPTS
    p = {k: 0.15 for k in cand}
    for t in tok:
        for table, w in ((TRACES, 1.0), (TRAPS, 1.3)):
            if table.get(t) in p:
                p[table[t]] += w
    if temperature <= 0.0:
        k = max(sorted(p), key=lambda a: p[a])
    else:
        e = [math.exp(p[a] / temperature) for a in sorted(p)]
        k = sorted(p)[pick(r(), [v / sum(e) for v in e])]
    y = k if "only" in tok else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k)
    out = tokenize(y)
    if len(out) > max_output:
        out = out[:max_output]
        y = " ".join(out)
    return {"status": "ok", "input": len(tok), "output": len(out), "response": y}


def read_label(y):
    for s in y.lower().split():
        if s in TOPICS:
            return s
    return None


TEST = TICKETS[48:]                              # test set: 72 records


def split_text(text, pieces):
    return tokenize(text) if pieces else text.lower().split()


def train_model(dataset, pieces=True, min_freq=1, smoothing=0.5):   # LEARNING procedure: count table
    counts, freq = {}, {}
    for x in dataset:
        freq[x["topic"]] = freq.get(x["topic"], 0) + 1
        for t in split_text(x["text"], pieces):
            c = counts.setdefault(t, {})
            c[x["topic"]] = c.get(x["topic"], 0) + 1
    majority = max(sorted(freq), key=lambda k: freq[k])

    def predict(text):
        p = {k: 0.0 for k in TOPICS}
        for t in split_text(text, pieces):
            c = counts.get(t, {})
            if sum(c.values()) < min_freq:
                continue
            for k in TOPICS:
                p[k] += math.log((c.get(k, 0) + smoothing) / (freq.get(k, 1) + 5 * smoothing))
        return max(sorted(p), key=lambda k: p[k]) if any(p.values()) else majority
    return predict


def hit_rate_train(n=48, **kw):
    f = train_model(TICKETS[:n], **kw)
    return sum(f(x["text"]) == x["topic"] for x in TEST) / len(TEST)


print(f"{'TRAINING approach setting':<31} {'A':>7} {'B':>7} {'move':>8}")
moves_train = []
for name, a, b in (("training examples 8 / 48", {"n": 8}, {"n": 48}),
                   ("token unit word / piece", {"pieces": False}, {"pieces": True}),
                   ("min frequency 3 / 1", {"min_freq": 3}, {"min_freq": 1}),
                   ("smoothing 0.05 / 0.50", {"smoothing": 0.05}, {"smoothing": 0.5})):
    va, vb = hit_rate_train(**a), hit_rate_train(**b)
    moves_train.append(abs(vb - va))
    print(f"{name:<31} {va:>7.4f} {vb:>7.4f} {vb - va:>+8.4f}")
print(f"settings that move it by more than 0.0200: {sum(v > 0.02 for v in moves_train)}/{len(moves_train)}, "
      f"labeled examples needed 48, what changes: the count table itself")

TRAINING approach setting             A       B     move
training examples 8 / 48         0.6250  1.0000  +0.3750
token unit word / piece          0.9861  1.0000  +0.0139
min frequency 3 / 1              0.8611  1.0000  +0.1389
smoothing 0.05 / 0.50            0.9861  1.0000  +0.0139
settings that move it by more than 0.0200: 2/4, labeled examples needed 48, what changes: the count table itself

Two of the four settings move the number above the threshold, and the largest move comes from the number of labeled examples: 0.6250 with eight examples, 1.0000 with forty-eight. This is both the strength and the bill of the training approach. The most effective way to raise the number is more labeled examples, and labeled examples are something that has to be produced. The remaining three settings also look inside the model: which unit it counts with, which evidence is thrown away, how sparse observations are smoothed. All of them are within reach and all of them change the model itself.

The Settings in the Using Approach’s Hands

In the using approach no setting looks inside the model. There is no count table, no weights, no training set. What remains is the surroundings of the call.

INSTRUCTION = ("Assign the ticket to one of these labels: invoice , fault , meter , tariff , "
               "outage . Write only the label as the response . Ticket: ")


def measure_use(instruction=INSTRUCTION, temperature=0.6, seed=SEED, retries=0):
    COUNTER[0] = 0
    hits = tokens = 0
    for x in TEST:
        e = None
        for j in range(retries + 1):
            s = call(instruction + x["text"], seed=seed + j, temperature=temperature)
            tokens += s["input"] + s["output"]
            e = read_label(s["response"])
            if e is not None:
                break
        hits += e == x["topic"]
    return hits / len(TEST), COUNTER[0], tokens


print(f"{'USING approach setting':<31} {'A':>7} {'B':>7} {'move':>8} "
      f"{'calls':>6} {'tokens':>9}")
moves_use = []
for name, a, b in (("label set no / yes",
                    {"instruction": "Answer this support ticket: "}, {}),
                   ("format constraint no / yes",
                    {"instruction": INSTRUCTION.replace("only the label", "an answer")}, {}),
                   ("temperature 0.6 / 0.0", {"temperature": 0.6}, {"temperature": 0.0}),
                   ("retries 0 / 1", {"retries": 0}, {"retries": 1}),
                   ("seed 20260218 / 20260219", {"seed": SEED}, {"seed": SEED + 1})):
    (va, ca, ba), (vb, cb, bb) = measure_use(**a), measure_use(**b)
    moves_use.append(abs(vb - va))
    print(f"{name:<31} {va:>7.4f} {vb:>7.4f} {vb - va:>+8.4f} {cb - ca:>+6d} {bb - ba:>+9d}")
print(f"settings that move it by more than 0.0200: {sum(v > 0.02 for v in moves_use)}/{len(moves_use)}, "
      f"labeled examples needed 0, what changes: everything outside the model")

USING approach setting                A       B     move  calls    tokens
label set no / yes               0.5417  0.7778  +0.2361     +0     +1143
format constraint no / yes       0.5000  0.7778  +0.2778     +0      -917
temperature 0.6 / 0.0            0.7778  0.9028  +0.1250     +0        -7
retries 0 / 1                    0.7778  0.7917  +0.0139     +2      +111
seed 20260218 / 20260219         0.7778  0.8194  +0.0417     +0       -14
settings that move it by more than 0.0200: 4/5, labeled examples needed 0, what changes: everything outside the model

The largest move comes from the format constraint: from 0.5000 to 0.7778, an increase of 0.2778. This row also has by far the largest minus sign in the cost column — it saves 917 tokens, while the temperature and seed rows shift the cost by only 7 and 14 tokens. Telling the model to write “only the label” both raises the hit rate and lowers the cost, because a short response is cheap. Writing the label set into the prompt also gains 0.2361, but its cost is 1143 tokens.

The fourth row pays off the second claim of this curriculum for the first time. Retrying moves the number by 0.0139, one record out of 72 and below the threshold, and in return asks for two extra calls and 111 tokens. In the plain scaffolding there was already no unreadable response left except the requests that failed at the surface; retrying could only rescue those, and it rescued one of the two. Adding scaffolding does not always improve enough to matter, but its cost is always certain.

The fifth row is not a setting. The seed is not your setting; it is the simulator’s uncertainty, and in this table it moves the number by 0.0417. So something that is not in your hands moves the number more than one of the four settings that are. Part of the effort spent on raising a product’s number is spent without knowing that the measurement itself moves by this much.

Who Holds Which Decision

No item of the two lists is shared. The table below writes this separation at the level of decisions.

The last row is the boundary of responsibility itself. In the training approach the address of a degradation is known: the data or the model. In the using approach the model does not degrade, because the model is not yours; what degrades is either the prompt, or the procedure that reads the output, or a change underneath the component. In the third case only one thing is left in your hands: having measured the surface. A measured surface tells you when the change happened; an unmeasured surface tells you only the complaint.

The difference between 1.0000 and 0.7778 among the numbers is also read in this frame. The training approach gave the higher number, but in return it asked for 48 labeled examples. The using approach gave 0.7778 without seeing any labeled example at all. On a task with a closed label set and plenty of examples, the first wins; when the label set is changeable, examples are expensive, or the task requires producing text, the question flips. What is being compared is not two numbers but two cost units.

Summary

• Two of the training approach’s four settings and three of the using approach’s four settings move the number by more than 0.0200; counting the seed, four of the using side’s five rows do. No item of the two lists is shared.
• The training approach’s strongest setting is the number of labeled examples: 0.6250 with 8 examples, 1.0000 with 48. The using approach gives 0.7778 with zero labeled examples.
• The using approach’s strongest setting is the format constraint: +0.2778 and a 917-token saving, by far the largest minus sign in the cost column.
• Retrying moves the number by 0.0139 and adds two calls and 111 tokens: adding scaffolding does not always improve enough to matter, but its cost is always certain.
• The seed, which is out of one’s hands, moves the number by 0.0417 — more than one of the four settings that are in one’s hands.
• The boundary of responsibility is read in the last row: when the number drops, the training approach changes the model and the using approach changes the scaffolding.

Next Step

In this lesson the seed row appeared once, with a single number, and was left there. Yet that row is the hardest design question of a product: the same user, on the same screen, sending the same text twice can get two different responses. The change of the overall hit rate from 0.7778 to 0.8194 is only the overall view of this; in how many records the response really changed and which part of that change is visible to the user was never asked. The next lesson counts the two runs of the same input record by record and puts three design decisions that narrow the uncertainty — offering options, imposing a format, grounding — side by side as hit rate and cost.