Natural Language Processing Metrics

In the previous lesson three metrics gave three different rankings, and the procedure that copies the input got 1.0000 on the recall side. That finding is not a property of a single task. In the eight lessons of this topic, metrics were read at four different levels: at the token level, at the entity level, at the document level, and through overlap. The same system gives four different numbers at these four levels, and the numbers do not preserve each other’s ranking.

This lesson is the close of the topic and finishes with a single setup: four systems, a single output format, four metrics. The output is one label per token for each note; all four of the metrics are read from the same labeling. The definitions of precision, recall and the balanced form were set up in evaluation-and-tuning of the Supervised Learning course and are not repeated here; the question here is not the definition but how the decision unit and the baseline change the metric.

• NL64. The text fiction is the same as that of the previous lessons and is fiction; seed 20260218.
• NL65. The gold labeling is fiction: all tokens of the phrases that come from the kind of the note itself are labeled with that kind, and all remaining tokens are labeled none. The borrowed side record counts as none — the event the note reports is not that one.
• NL66. The output format is single: each system produces one label for each token. All four metrics are computed from this single output; no system gives a separate output for a metric.
• NL67. A span is the longest run of consecutive tokens with the same label. At the entity level a span counts as correct only if its start, its end and its label all match.
• NL68. The document-level decision is derived from the output: the most frequent label among the marked tokens counts as the kind of the note; if there is no mark at all, the most frequent class is written.
• NL69. The overlap metric is the recall side of the previous lesson: how many of the tokens that are marked in the gold labeling are also marked in the output. Repeated tokens are counted by clipping.
• NL70. The four metrics have four different baselines, and each baseline is the dumbest procedure of that metric: writing the most frequent label to every token, marking no span at all, writing the most frequent class, marking the whole input.
• NL71. The tables the systems learn are built from the training set only; the held-out set is the last 500 notes.
• NL72. None of the four systems is derived from another; all four run on the same held-out set, with the same output format.

One Output, Four Readings

Once the gold labeling is set up, two baselines can be read directly: the procedure that writes the most frequent label to every token at the token level, and the procedure that writes the most frequent class at the document level.

# FICTIONAL field notes — an extended version of the note field of the Data Preparation and Feature Engineering course. Seed 20260218.

SEED, M32 = 20260218, 0xFFFFFFFF
KINDS = ("ordinary", "unreachable", "meter fault", "illicit use")
SHARES = (0.46, 0.22, 0.19, 0.13)
PHRASES = {t: s.split("|") for t, s in zip(KINDS, (
    "okuma normal|okuma normaldi|sayac erisimi saglandi|sayaca erisildi|olcum tekrarlandi|"
    "olcumu yineledik|vana kontrol edildi|vanada sorun yok|kapak temiz|kapakta iz yok|"
    "gosterge okunakli|gostergede rakam net",
    "abone evde degildi|abone bulunamadi|erisim saglanamadi|erisim yok|kapi zili calismiyor|"
    "zil calismadi|bahce kapisi kilitli|bahce kapisi kapali|bodrum katina inilemedi|"
    "bodrum kilitli|adres bulunamadi|adres eksik",
    "gosterge donmuyor|gostergede hareket yok|sayac cami kirik|sayacin cami catlak|"
    "rakam okunmuyor|rakamlar silik|ibre sabit kalmis|ibre kilitli|mekanizma takilmis|"
    "mekanizmada takilma|sayac durmus|sayac calismiyor",
    "muhur izi var|muhurde iz goruldu|muhur kopmus|muhurler kopuk|baglanti degistirilmis|"
    "baglantida degisiklik|kelepce gevsek|kelepcede gevseme|boru ek yeri yeni|boruda yeni ek|"
    "sayac ters takilmis|sayaci ters takmislar"))}
NOISE = ("arac park halinde|kapak kirli|hava yagisli|adres teyit edildi|not kisa tutuldu|"
         "ikinci ziyaret|bahce icinde|kaldirim uzerinde").split("|")
REGIONS = "kuzey hatti|guney hatti|dogu hatti|bati hatti|merkez hatti".split("|")
PHRASE_KIND = {c: t for t in KINDS for c in PHRASES[t]}


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def build_note(n):
    r = rng(SEED + 41000 + n)
    u, k = r(), 0.0
    for i, p in enumerate(SHARES):
        k += p
        if u < k:
            t = KINDS[i]
            break
    p = [PHRASES[t][int(r() * 12)]]
    if r() < 0.55:
        p.append(PHRASES[t][int(r() * 12)])
    if r() < 0.34:
        p.append(PHRASES[KINDS[int(r() * 4)]][int(r() * 12)])
    if r() < 0.62:
        p.append(NOISE[int(r() * 8)])
    if r() < 0.30:
        p.append(REGIONS[int(r() * 5)])
    return {"text": ", ".join(p), "kind": t, "phrases": p,
            "summary": ", ".join(c for c in p if PHRASE_KIND.get(c) == t)}
NONE = "none"


def tokenize(m):
    return [w.strip(",").lower() for w in m.split() if w.strip(",")]


DATA = []
for i in range(1600):
    x = build_note(i)
    z, e = [], []
    for c in x["phrases"]:                   # GOLD label: phrases of its own kind
        w = tokenize(c)
        z += w
        e += [x["kind"] if PHRASE_KIND.get(c) == x["kind"] else NONE] * len(w)
    x["tokens"], x["gold"] = z, e
    DATA.append(x)
TRAIN, HELD_OUT = DATA[:1100], DATA[1100:]
label_counts = {}
for x in TRAIN:
    for u in x["gold"]:
        label_counts[u] = label_counts.get(u, 0) + 1
MOST_COMMON_LABEL = max(label_counts, key=label_counts.get)
MOST_COMMON_KIND = max(KINDS, key=lambda t: sum(y["kind"] == t for y in TRAIN))
n = len(HELD_OUT)
BASE_TOKEN = (sum(u == MOST_COMMON_LABEL for x in HELD_OUT for u in x["gold"])
              / sum(len(x["gold"]) for x in HELD_OUT))
BASE_DOC = sum(x["kind"] == MOST_COMMON_KIND for x in HELD_OUT) / n
print(f"held out {n} notes, {sum(len(x['tokens']) for x in HELD_OUT)} tokens, "
      f"marked tokens {sum(u != NONE for x in HELD_OUT for u in x['gold'])}")
total_labels = sum(label_counts.values())
print("gold label distribution (train):  " + "  ".join(
    f"{u}={label_counts[u] / total_labels:.3f}"
    for u in sorted(label_counts, key=lambda u: -label_counts[u])[:3]))
print("                                  " + "  ".join(
    f"{u}={label_counts[u] / total_labels:.3f}"
    for u in sorted(label_counts, key=lambda u: -label_counts[u])[3:]))
print(f"token-level baseline ('{MOST_COMMON_LABEL}' on every token) {BASE_TOKEN:.4f}, "
      f"document-level baseline ('{MOST_COMMON_KIND}') {BASE_DOC:.4f}")

held out 500 notes, 3305 tokens, marked tokens 1993
gold label distribution (train):  none=0.413  ordinary=0.267  unreachable=0.125
                                  meter fault=0.111  illicit use=0.084
token-level baseline ('none' on every token) 0.3970, document-level baseline ('ordinary') 0.4420

The label distribution says why the token-level baseline is 0.3970: about two fifths of the tokens belong to no event, and none alone is the most frequent label. The Supervised Learning course had measured how misleading overall accuracy is on an imbalanced class set; here the same imbalance appears the moment the decision unit of the metric is chosen. Of the 3,305 tokens, 1,993 are marked and 1,312 are not.

Four Systems, Four Rankings

The four systems represent four different design decisions. A — phrase-level selector makes the decision over the phrase: if the rate at which a phrase entered the gold labeling in the training set is above 0.75, all tokens of the phrase are marked, and the label comes from the document prediction. B — token-level selector makes the decision over single tokens: if the marking rate of a word is above 0.40, it is marked. C — narrow marker marks only the first phrase and takes the label from the kind of that phrase itself. D — mark everything writes the most frequent kind to every token; it learns nothing.

PHRASE_COUNTS, TOKEN_COUNTS, KIND_SCORES = {}, {}, {}
for x in TRAIN:
    for c in x["phrases"]:
        a = PHRASE_COUNTS.setdefault(c, [0, 0])
        a[0] += 1
        a[1] += PHRASE_KIND.get(c) == x["kind"]
        if c in PHRASE_KIND:
            KIND_SCORES.setdefault(c, {})
            KIND_SCORES[c][x["kind"]] = KIND_SCORES[c].get(x["kind"], 0) + 1
    for w, u in zip(x["tokens"], x["gold"]):
        a = TOKEN_COUNTS.setdefault(w, [0, 0])
        a[0] += 1
        a[1] += u != NONE
PHRASE_RATE = {c: a[1] / a[0] for c, a in PHRASE_COUNTS.items()}
TOKEN_RATE = {w: a[1] / a[0] for w, a in TOKEN_COUNTS.items()}


def predict_kind(x):
    s = {}
    for c in x["phrases"]:
        for t, v in KIND_SCORES.get(c, {}).items():
            s[t] = s.get(t, 0) + v
    return max(s, key=s.get) if s else MOST_COMMON_KIND


def system_phrase(x):
    t, e = predict_kind(x), []
    for c in x["phrases"]:
        e += [(t if PHRASE_RATE.get(c, 0.0) > 0.75 else NONE)] * len(tokenize(c))
    return e


def system_token(x):
    t = predict_kind(x)
    return [(t if TOKEN_RATE.get(w, 0.0) > 0.40 else NONE) for w in x["tokens"]]


def system_narrow(x):
    e = []
    for i, c in enumerate(x["phrases"]):
        e += [(PHRASE_KIND.get(c, NONE) if i == 0 else NONE)] * len(tokenize(c))
    return e


def system_all(x):
    return [MOST_COMMON_KIND] * len(x["tokens"])


def spans(e):
    a, i = [], 0
    while i < len(e):
        if e[i] == NONE:
            i += 1
            continue
        j = i
        while j + 1 < len(e) and e[j + 1] == e[i]:
            j += 1
        a.append((i, j, e[i]))
        i = j + 1
    return a


def measure(f):
    token_hit = token_total = span_hit = span_pred = span_gold = doc_hit = 0
    overlap_sum = 0.0
    for x in HELD_OUT:
        e = f(x)
        token_hit += sum(u == v for u, v in zip(e, x["gold"]))
        token_total += len(e)
        g, p = spans(x["gold"]), spans(e)
        span_hit += sum(u in g for u in p)
        span_pred += len(p)
        span_gold += len(g)
        s = {}
        for u in e:
            if u != NONE:
                s[u] = s.get(u, 0) + 1
        doc_hit += (max(s, key=s.get) if s else MOST_COMMON_KIND) == x["kind"]
        left = [w for w, u in zip(x["tokens"], x["gold"]) if u != NONE]
        hit = 0
        for w in [w for w, u in zip(x["tokens"], e) if u != NONE]:
            if w in left:
                left.remove(w)
                hit += 1
        overlap_sum += hit / max(1, sum(u != NONE for u in x["gold"]))
    pr, rc = span_hit / max(1, span_pred), span_hit / max(1, span_gold)
    return (token_hit / token_total, 2 * pr * rc / (pr + rc) if pr + rc else 0.0,
            doc_hit / len(HELD_OUT), overlap_sum / len(HELD_OUT))


SYSTEMS = (("A", "phrase-level selector", system_phrase),
           ("B", "token-level selector", system_token),
           ("C", "narrow marker", system_narrow), ("D", "mark everything", system_all))
SCORES = {k: measure(f) for k, _, f in SYSTEMS}
METRICS = ("token", "entity", "document", "overlap")
print(f"{'':<3}{'system':<24}" + "".join(f"{o:>11}" for o in METRICS))
for k, name, _ in SYSTEMS:
    print(f"{k:<3}{name:<24}" + "".join(f"{v:>11.4f}" for v in SCORES[k]))
for j, o in enumerate(METRICS):
    print(f"{o:<9} ranking: " + " > ".join(
        sorted(SCORES, key=lambda a: -SCORES[a][j])))

   system                        token     entity   document    overlap
A  phrase-level selector        0.8136     0.6779     0.8320     0.8423
B  token-level selector         0.8378     0.6036     0.9320     0.9693
C  narrow marker                0.7619     0.4000     1.0000     0.6901
D  mark everything              0.2732     0.0920     0.4420     1.0000
token     ranking: B > A > C > D
entity    ranking: A > B > C > D
document  ranking: C > B > A > D
overlap   ranking: D > B > A > C

The four ranking lines are the answer of this lesson. All four metrics read the same four outputs, and in all four the ranking is different: B > A > C > D, A > B > C > D, C > B > A > D, D > B > A > C. The first of each of the four metrics is also a different system. The question “which system is better” has no answer in this setup; the question that has an answer is “on which metric”.

Where the differences come from can be read one by one. The difference between A and B lies in the decision unit: because B decides per token, it leads at the token level with 0.8378, but because it makes its decision independently of its neighbor, it skips a word in the middle of a phrase and splits the span in two; at the entity level it falls to 0.6036, 0.0743 below A. The same output, the same number of correct tokens, two different rankings.

C is the system that marks the fewest tokens among all systems, and it gets 1.0000 at the document level. This number is a consequence of the fiction — the phrase that reports the decision is written at the beginning of the note — and cannot be read as a superiority of method. The same system stays at 0.4000 at the entity level, because in a part of the notes the gold span covers two phrases and C never marks the second.

The D row is the third payment, in this topic, of the second claim of the course. D learns nothing: it writes the most frequent kind of the training set to every token. At the token level it falls even below the baseline with 0.2732, at the entity level it gets 0.0920 — and on the recall side of the overlap metric it becomes the first of the four systems with 1.0000. If a metric can put a procedure that has learned nothing in first place, printing that metric alone is not a measurement but a choice.

Each Metric’s Own Baseline

The four metrics have four different baselines, and the baselines are not on the same scale. The size of a number says nothing by itself; what says something is how far that number stands from its own baseline.

BASELINES = (("token-level accuracy", "write the most frequent label to every token", BASE_TOKEN),
             ("entity-level balanced form", "mark no span", 0.0),
             ("document-level accuracy", "write the most frequent class", BASE_DOC),
             ("overlap recall side", "mark the whole input", 1.0))
print(f"{'metric':<28}{'baseline procedure':<46}{'baseline':>9}{'best':>8}{'difference':>11}")
for j, (o, procedure, base) in enumerate(BASELINES):
    best = max(SCORES, key=lambda a: SCORES[a][j])
    print(f"{o:<28}{procedure:<46}{base:>9.4f}{SCORES[best][j]:>8.4f}"
          f"{SCORES[best][j] - base:>+11.4f}")

metric                      baseline procedure                             baseline    best difference
token-level accuracy        write the most frequent label to every token     0.3970  0.8378    +0.4408
entity-level balanced form  mark no span                                     0.0000  0.6779    +0.6779
document-level accuracy     write the most frequent class                    0.4420  1.0000    +0.5580
overlap recall side         mark the whole input                             1.0000  1.0000    +0.0000

The table gives four different distances in four rows. At the entity level the best system is 0.6779 above the baseline, at the document level 0.5580, at the token level 0.4408. These three rows can be ranked but cannot be compared: three different decision units, three different baselines, three different scales.

The last row is the closing sentence of the topic. On the recall side of the overlap metric the baseline is 1.0000, because the procedure that marks the whole input necessarily covers every token of the gold labeling. The best system also gets 1.0000, and the difference is exactly 0.0000. This metric has a baseline that nothing can be added on top of: no learning can pass it, only tie it. The first question to ask when choosing a metric is what the dumbest procedure of that metric gets on the same metric; the second is whether any room is left above that number to pass.

The baseline of 0.0000 at the entity level, on the other hand, is at the opposite end and it also misleads: a procedure that marks no span gets zero, so passing the baseline on this metric is free. The more honest baseline is system D in the same table: the procedure that counts the whole note as a single span gets 0.0920, and the difference of the best system drops from 0.6779 to 0.5859.

Summary

• A single token labeling can be read at four different levels, and the four metrics produce four different numbers from the same output.
• The four metrics give four different rankings, and the first of the four is four different systems: B at the token level, A at the entity level, C at the document level, D on overlap.
• The decision unit determines the metric: the system that decides per token leads at the token level and falls 0.0743 behind at the entity level because it splits the span.
• The system that learns nothing falls below the baseline at the token level but becomes first with 1.0000 on the recall side of overlap.
• Each metric has its own baseline, and the baselines are not on the same scale: 0.3970, 0.0000, 0.4420 and 1.0000; on the last one the difference between the baseline and the best system is 0.0000.

Next Step

In eight lessons the input was always a sequence. The tokens were ordered from left to right, the neighbor of a word lay on a single axis, a span was defined by a start and an end index, and the order itself carried information: “muhur izi var” (seal mark present) and “var izi muhur” (present mark seal) have the same bag of words but are not the same thing. That the overlap metric differs on 2-grams from single words was also a result of this single axis.

In a domain where the input is two-dimensional and the neighborhood opens not to the right and left but in every direction, every part of this setup is asked again. A span is no longer a start and an end but a rectangle; overlap is no longer a counter but a ratio of areas; there is no such thing as a vocabulary, because the unit of the input is not a word but a number, and that number has no meaning on its own. The next topic begins with images and opens with the same question: what is the dumbest procedure of this domain, and what does that procedure get on its own metric.