The Distinction Between AI and General AI

The previous three lessons treated the model as a given component, measured it from its surface, counted its settings, and translated its uncertainty into product language. The words used in those three lessons are also a design object. “Capability”, “understands”, “learns”, and “general intelligence” are things too different to be written into the same table: some of them can be tied to a test and give a number, and some can never be falsified under any condition.

This lesson ties that separation to a procedure. A series of fictional claim sentences is parsed into four components, the testable ones are selected, and they are actually run. What is measured is not the truth of the claims but how many of them can be tied to a test. At the end of the lesson one of them is tied, run, and turns out wrong; this shows that making a claim measurable does not make it true.

• FR23. A claim counts as testable when it carries four components: a task, a metric, a set, and a threshold together with its direction. If one of the four is missing, the claim counts as general. This is a decision and it is held fixed throughout the lesson.
• FR24. The components are read from the tokens of the sentence; the classification is not done by hand, it is a running procedure. The tokenizer carries over unchanged from the previous lessons.
• FR25. The claim sentences are fiction and none of them is about a product, a model, or an institution; all of them are about the fictional support assistant built in the previous three lessons.
• FR26. The system under test is the strongest scaffolding of the previous lesson: a grounded instructed call. The measurement is made on the 72-record test set and in two runs.
• FR27. A test is run only when the classification finds the claim measurable. No test is written for general claims, because there is no test to write.
• FR28. The direction of the threshold is part of the claim. A metric without a threshold gives a number but makes no decision; a claim passes or fails only together with its direction.

When a Claim Is Testable

The sentence “the system understands support tickets” can never be falsified, because it does not say on which task, on which set, with which number, and at which threshold it will count as understood. The sentence “label hit rate on the 72 record set is above 0.90”, on the other hand, can be falsified. The difference between the two is not truth but tie-ability.

# SIMULATOR. This code does not call a real model endpoint; it is a language model
# simulator written with the standard library. The support tickets and claims are also
# FICTIONAL.
import math

SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW, MAX_OUTPUT = 240, 24


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(text):
    tok = []
    for s in text.lower().replace(",", " , ").split():
        while len(s) > 5:
            tok.append(s[:5])
            s = s[5:]
        tok.append(s)
    return tok


def trace_id(text):
    h = 2166136261
    for c in text:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


TOPICS = ["invoice", "fault", "meter", "tariff", "outage"]
TEMPLATES = {
    "invoice": ["the invoice amount this month is very high compared to last month",
                "i paid my invoice but the system shows a debt",
                "where can i see the invoice details"],
    "fault": ["the water is not flowing at home and there is a fault i think",
              "the connection is broken and it is not working at all",
              "i opened a fault ticket but the team never came"],
    "meter": ["the meter reading was entered wrong and the index does not match",
              "i want an appointment for the meter change",
              "the index on the meter and the invoice are not the same"],
    "tariff": ["i want to learn which tariff tier i am on",
               "what do i need to do for a tariff change",
               "does the unit price change by subscriber type"],
    "outage": ["there is an outage in the neighborhood when will it be back",
               "where is the planned outage announcement tracked",
               "we have been having outages since the evening"],
}
EXTRAS = ["the same issue happened last month too", "i am not satisfied with this",
          "i urgently need information", "i do not have my record number"]

TICKETS = []
for i in range(120):
    r = rng(SEED + i)
    k = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])]
    m = TEMPLATES[k][int(r() * 3)]
    if r() < 0.34:
        m += " " + EXTRAS[int(r() * 4)]
    TICKETS.append({"id": 4001 + i, "text": m, "topic": k})

CONCEPTS = TOPICS + ["payment", "complaint", "information", "general"]
TRACES = {"invoi": "invoice", "amoun": "invoice", "detai": "invoice",
          "fault": "fault", "broke": "fault", "worki": "fault", "flowi": "fault",
          "meter": "meter", "index": "meter", "readi": "meter",
          "tarif": "tariff", "unit": "tariff", "subsc": "tariff", "tier": "tariff",
          "outag": "outage", "annou": "outage", "neigh": "outage",
          "debt": "payment", "paid": "payment", "price": "payment",
          "satis": "complaint", "issue": "complaint", "urgen": "complaint",
          "infor": "information", "learn": "information", "where": "information"}
TRAPS = {"chang": "tariff", "recor": "fault", "high": "tariff", "shows": "invoice",
         "team": "outage", "track": "tariff", "enter": "invoice"}
FREE_FORM = ["your request seems to relate to {k}",
             "this falls under {k} , forwarded to the relevant team",
             "your record has been received , we will review the matter and get back to you"]
COUNTER = [0]


def call(prompt, seed=SEED, temperature=0.6, max_output=MAX_OUTPUT):
    COUNTER[0] += 1
    tok = tokenize(prompt)
    if len(tok) > WINDOW:
        return {"status": "context_overflow", "input": len(tok), "output": 0, "response": ""}
    if COUNTER[0] % 37 == 0:
        return {"status": "rate_limit", "input": len(tok), "output": 0, "response": ""}
    r = rng(seed ^ trace_id(prompt))
    if len(tok) > 90 and r() < 0.04:
        return {"status": "timeout", "input": len(tok), "output": 0, "response": ""}
    cand = [k for k in CONCEPTS if k[:5] in tok]
    if len(cand) < 3:
        cand = CONCEPTS
    p = {k: 0.15 for k in cand}
    for t in tok:
        for table, w in ((TRACES, 1.0), (TRAPS, 1.3)):
            if table.get(t) in p:
                p[table[t]] += w
    if temperature <= 0.0:
        k = max(sorted(p), key=lambda a: p[a])
    else:
        e = [math.exp(p[a] / temperature) for a in sorted(p)]
        k = sorted(p)[pick(r(), [v / sum(e) for v in e])]
    y = k if "only" in tok else FREE_FORM[pick(r(), [0.42, 0.26, 0.32])].format(k=k)
    out = tokenize(y)
    if len(out) > max_output:
        out = out[:max_output]
        y = " ".join(out)
    return {"status": "ok", "input": len(tok), "output": len(out), "response": y}


def read_label(y):
    for s in y.lower().split():
        if s in TOPICS:
            return s
    return None


CLAIMS = [("label hit rate on the 72 record set is above 0.90", "hit_rate", 0.90, ">"),
          ("the system understands support tickets", "", 0, ""),
          ("the share of changed decisions on the 72 record set is below 0.10", "change", 0.10, "<"),
          ("the system is a smart assistant", "", 0, ""),
          ("the system always produces the correct label", "", 0, ""),
          ("the system has human level language understanding", "", 0, ""),
          ("the share of responses without a label on the 72 record set is below 0.05", "unlabeled", 0.05, "<"),
          ("the system has general intelligence", "", 0, ""),
          ("the system stays below 140 tokens per request", "", 0, ""),
          ("the share of correct documents retrieved from the vocabulary is above 0.80", "retrieval", 0.80, ">"),
          ("the system keeps learning", "", 0, ""),
          ("tokens per request on the 72 record set is below 140", "tokens", 140, "<"),
          ("the system produces creative solutions", "", 0, ""),
          ("the system gets smarter as it grows", "", 0, "")]
TASK_KEYS = ("label", "decis", "answe", "retri", "reque")
METRIC_KEYS = ("hit", "share", "token", "call", "step")
SET_KEYS = ("recor", "set", "vocab", "examp")
DIRECTION_KEYS = ("above", "below")


def matches(pieces, keys):
    return any(t.startswith(k) for t in pieces for k in keys)


def classify(c):
    b = tokenize(c)
    return (matches(b, TASK_KEYS), matches(b, METRIC_KEYS), matches(b, SET_KEYS),
            matches(b, DIRECTION_KEYS) and any(any(d.isdigit() for d in t) for t in b))


print(f"{'claim':<76} {'task':>5} {'metric':>6} {'set':>4} {'threshold':>9} {'class':>11}")
CHECKS = []
for c, metric, threshold, direction in CLAIMS:
    g, o, k, e = classify(c)
    if g and o and k and e:
        CHECKS.append((metric, threshold, direction))
    print(f"{c:<76} {'+' if g else '-':>5} {'+' if o else '-':>6} {'+' if k else '-':>4} "
          f"{'+' if e else '-':>9} {'measurable' if g and o and k and e else 'general':>11}")
numeric = sum(1 for c, *_ in CLAIMS if classify(c)[3])
print(f"\ncan be tied to a test {len(CHECKS)}/{len(CLAIMS)} = {len(CHECKS) / len(CLAIMS):.4f}, "
      f"carries a number and direction yet cannot be tied {numeric - len(CHECKS)}")

claim                                                                         task metric  set threshold       class
label hit rate on the 72 record set is above 0.90                                +      +    +         +  measurable
the system understands support tickets                                           -      -    -         -     general
the share of changed decisions on the 72 record set is below 0.10                +      +    +         +  measurable
the system is a smart assistant                                                  -      -    -         -     general
the system always produces the correct label                                     +      -    -         -     general
the system has human level language understanding                                -      -    -         -     general
the share of responses without a label on the 72 record set is below 0.05        +      +    +         +  measurable
the system has general intelligence                                              -      -    -         -     general
the system stays below 140 tokens per request                                    +      +    -         +     general
the share of correct documents retrieved from the vocabulary is above 0.80       +      +    +         +  measurable
the system keeps learning                                                        -      -    -         -     general
tokens per request on the 72 record set is below 140                             +      +    +         +  measurable
the system produces creative solutions                                           -      -    -         -     general
the system gets smarter as it grows                                              -      -    -         -     general

can be tied to a test 5/14 = 0.3571, carries a number and direction yet cannot be tied 1

Of the fourteen sentences, five can be tied to a test; the share is 0.3571. Of the remaining nine, eight carry none of the four components or only one. The fifth row is the most instructive of them: “the system always produces the correct label” carries a task name but gives no metric, no set, and no threshold. “Always” is not a threshold; because it is not written on which set it will be tested, a single counterexample refutes it, and for that reason nobody tests it.

The number in the last line gives the real distinction: a claim can carry a number and still be untestable. The ninth sentence says “the system stays below 140 tokens per request”; it has a task, a metric, a threshold, and a direction, but no set. Because it is not written on which tickets, with which scaffolding, and with prompts of which length it was measured, the number can be produced to take the desired value. The twelfth row adds the set to the same claim and the sentence becomes testable. The only difference between them is the phrase on the 72-record set.

The Tests Are Run

The classification says that a sentence is testable, not that it is true. The second block runs five tests on the strongest scaffolding of the previous lesson.

DOCUMENTS = [("TR-11", "the tariff tier is determined by subscriber type and unit price"),
             ("TR-12", "tariff change requests are accepted at the start of each period"),
             ("FT-21", "the invoice amount is calculated from the period reading , "
                       "the detail is on screen"),
             ("FT-22", "after payment the debt entry is closed the same day"),
             ("SY-31", "meter index reading is taken each period and goes into the log"),
             ("SY-32", "meter change is done by appointment , the old index stays in the log"),
             ("AR-41", "when a fault ticket is opened the field crew is dispatched"),
             ("AR-42", "for a broken connection the line is checked first"),
             ("KS-51", "the planned outage announcement is published first"),
             ("KS-52", "outage follow-up per neighborhood is done on a map")]
CODES = {"TR": "tariff", "FT": "invoice", "SY": "meter", "AR": "fault", "KS": "outage"}
TEST = TICKETS[48:]
INSTRUCTION = ("Assign the ticket to one of these labels: invoice , fault , meter , tariff , "
               "outage . Write only the label as the response . Ticket: ")


def retrieve(text):
    b = set(tokenize(text))
    return max(DOCUMENTS, key=lambda d: (len(b & set(tokenize(d[1]))), d[0]))


def run(seed):
    COUNTER[0] = 0
    labels, tokens, docs = [], 0, 0
    for x in TEST:
        k, m = retrieve(x["text"])
        docs += CODES[k[:2]] == x["topic"]
        s = call(INSTRUCTION + "Document " + k + ": " + m + " . Ticket: " + x["text"],
                 seed=seed)
        tokens += s["input"] + s["output"]
        labels.append(read_label(s["response"]))
    return labels, tokens, docs / len(TEST)


e1, b1, doc_rate = run(SEED)
e2, b2, _ = run(SEED + 1)
n = len(TEST)
MEASURES = {"hit_rate": sum((a == x["topic"]) + (b == x["topic"])
                            for a, b, x in zip(e1, e2, TEST)) / (2 * n),
            "change": sum(a != b for a, b in zip(e1, e2)) / n,
            "unlabeled": sum((a is None) + (b is None) for a, b in zip(e1, e2)) / (2 * n),
            "retrieval": doc_rate, "tokens": (b1 + b2) / (2 * n)}

print(f"{'test':<12} {'threshold':>12} {'measured':>10} {'result':>8}")
passed = 0
for metric, threshold, direction in CHECKS:
    v = MEASURES[metric]
    ok = v > threshold if direction == ">" else v < threshold
    passed += ok
    b = "{:.4f}" if threshold < 1 else "{:.1f}"
    print(f"{metric:<12} {direction + ' ' + b.format(threshold):>12} {b.format(v):>10} "
          f"{'passed' if ok else 'failed':>8}")
print(f"\ntested {len(CHECKS)}, passed {passed}, failed {len(CHECKS) - passed}; "
      f"tested among general claims 0")

test            threshold   measured   result
hit_rate         > 0.9000     0.8264   failed
change           < 0.1000     0.0972   passed
unlabeled        < 0.0500     0.0139   passed
retrieval        > 0.8000     0.9028   passed
tokens            < 140.0       75.2   passed

tested 5, passed 4, failed 1; tested among general claims 0

Four of the five tests passed, one failed. The first row is the most important result of the lesson: the claim said 0.90, the measurement gave 0.8264. The claim is wrong. A measurable claim can turn out wrong; what makes it valuable is not that it is true but that it can be wrong. The third row shows the other side: the claim that the unlabeled response share stays below 0.05 passed at 0.0139, and a claim that passes is also a result, not a courtesy. The measurement covered the whole surface, including the calls that hit the rate limit and produced no response, and the claim still held.

The last line is the silent result of the lesson. Among the general claims, the number tested is zero. Nothing could be said about nine sentences: they neither passed nor failed. The claim that a system “has general intelligence” or “gets smarter as it develops” cannot be discussed, because no counterexample can be produced for it; it has no place in an engineering document, because it changes no decision. The term general artificial intelligence (general AI) is not a capability class but a lack of tests. A sentence that wants to say what a system does has to write its task, its set, its metric, and its threshold; when it does not write them, what it says cannot be measured.

Summary

• A claim becomes testable with four components: a task, a metric, a set, and a threshold together with its direction. Of the fourteen fictional sentences, five carry these four; the share is 0.3571.
• Carrying a number is not enough. The sentence “the system stays below 140 tokens per request” carries a task, a metric, and a threshold but has no set; when “on the 72-record set” is added to the same sentence, it becomes testable.
• Five tests were run: four passed, one failed. The decision change share of 0.0972, the unlabeled response share of 0.0139, the document retrieval share of 0.9028, and the tokens per request of 75.2 passed their thresholds.
• A measurable claim can also turn out wrong: the label hit rate claim said 0.90, the measurement gave 0.8264. The measurement covers the whole surface, rate limit included.
• Among the general claims, the number tested is zero. The term general AI is not a capability class but a lack of tests; in an engineering document it changes no decision.

Next Step

In four lessons the model was always looked at from the outside. Its measure was set up, who holds its settings was counted, its uncertainty was translated into product language, and the sentences made about it were separated by testability. All of this was done on the surface of a closed box, and that was a deliberate choice. But every number measured on the surface has a reason, and that reason is inside: why 0.8264 is not 1.0000, why “meter replacement” and “tariff change” get confused, why a seven-word ticket makes twelve tokens, why the window ends at 240. The next topic opens with these questions: what is the unit the model works in, how does it use distant context, where does its limit come from, and in which steps is a response produced.