Text Classification

Over four lessons the text was turned into numbers. How the tokenization decision moved the size of the vocabulary, what the bag of words and term weighting added to hand-counted features, how the embedding is learned from context and how the labels are distributed to tokens were measured. In all these measurements the decision unit was the token: every token received its own label, and the metric was read at the token or the entity level.

Giving a single decision to the whole of a document was never tried. A field note is read and one of four event kinds is written: meter fault, suspected illicit use, unreachable, ordinary. In this setup the output is a single label and the length of the input is variable. The question of this lesson is this: which step of the pipeline that produces this decision adds how many points. There are three steps — tokenization, term weighting, classifier — and the contribution of each is not independent of the others.

• NL41. The field notes are fiction. They are an extended version of the note field produced by feature-selection/02 of the Data Preparation and Feature Engineering course: every note is built from an event phrase, sometimes a side record borrowed from another kind, a noise phrase and a line marker. Seed 20260218.
• NL42. The event kind label is the kind the note was generated from; it is not derived from the words inside the note. Because of the borrowed side record the text sometimes points to another kind; this pulls the ceiling of the measure down and is a deliberate part of the fiction.
• NL43. The set is 1,600 notes; the first 1,100 are the training set and the last 500 the held-out set.
• NL44. The domain baseline is the most frequent class. In a four-class task random assignment stays around 0.25; the most frequent class is higher than that, and since the classes are imbalanced it is the real comparison.
• NL45. Every step of the pipeline is learned from the training set only: the vocabulary, the document frequencies and the model weights. This rule was built in feature-selection/05 of the Data Preparation and Feature Engineering course and continues here.
• NL46. The vocabulary takes the words that occur in at least two notes in the training set.
• NL47. Both classifiers are written within the lesson and are the model: the class mean (cosine similarity to the mean vector of every class) and a four-class linear model.

A Single Decision for the Whole Document

The input of classification is a document and its output is a label. The difference from sequence labeling is not a difference of scale but a difference of decision unit: there was one output for every token there, here there is one output for the whole note, and how many tokens the note consists of does not change the shape of the output.

# FICTIONAL field notes — an extended version of the note field of the Data Preparation and
# Feature Engineering course. Seed 20260218.
import math

SEED, M32 = 20260218, 0xFFFFFFFF
KINDS = ("ordinary", "unreachable", "meter fault", "illicit use")
SHARES = (0.46, 0.22, 0.19, 0.13)
PHRASES = {k: s.split("|") for k, s in zip(KINDS, (
    "okuma normal|okuma normaldi|sayac erisimi saglandi|sayaca erisildi|olcum tekrarlandi|"
    "olcumu yineledik|vana kontrol edildi|vanada sorun yok|kapak temiz|kapakta iz yok|"
    "gosterge okunakli|gostergede rakam net",
    "abone evde degildi|abone bulunamadi|erisim saglanamadi|erisim yok|kapi zili calismiyor|"
    "zil calismadi|bahce kapisi kilitli|bahce kapisi kapali|bodrum katina inilemedi|"
    "bodrum kilitli|adres bulunamadi|adres eksik",
    "gosterge donmuyor|gostergede hareket yok|sayac cami kirik|sayacin cami catlak|"
    "rakam okunmuyor|rakamlar silik|ibre sabit kalmis|ibre kilitli|mekanizma takilmis|"
    "mekanizmada takilma|sayac durmus|sayac calismiyor",
    "muhur izi var|muhurde iz goruldu|muhur kopmus|muhurler kopuk|baglanti degistirilmis|"
    "baglantida degisiklik|kelepce gevsek|kelepcede gevseme|boru ek yeri yeni|boruda yeni ek|"
    "sayac ters takilmis|sayaci ters takmislar"))}
NOISE = ("arac park halinde|kapak kirli|hava yagisli|adres teyit edildi|not kisa tutuldu|"
         "ikinci ziyaret|bahce icinde|kaldirim uzerinde").split("|")
REGIONS = "kuzey hatti|guney hatti|dogu hatti|bati hatti|merkez hatti".split("|")
SUSPICION_P = {"illicit use": 0.86, "meter fault": 0.21, "unreachable": 0.07, "ordinary": 0.03}


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def build_note(n):
    r = rng(SEED + 41000 + n)
    u, k = r(), 0.0
    for i, p in enumerate(SHARES):
        k += p
        if u < k:
            t = KINDS[i]
            break
    p = [PHRASES[t][int(r() * 12)]]
    if r() < 0.55:
        p.append(PHRASES[t][int(r() * 12)])
    if r() < 0.34:                          # side record borrowed from another kind
        p.append(PHRASES[KINDS[int(r() * 4)]][int(r() * 12)])
    if r() < 0.62:
        p.append(NOISE[int(r() * 8)])
    if r() < 0.30:
        p.append(REGIONS[int(r() * 5)])
    return {"text": ", ".join(p), "kind": t, "suspicious": 1 if r() < SUSPICION_P[t] else 0}


NOTES = [build_note(i) for i in range(1600)]
TRAIN, HELD_OUT = NOTES[:1100], NOTES[1100:]
KIND_INDEX = {t: i for i, t in enumerate(KINDS)}
counts = {}
for x in TRAIN:
    counts[x["kind"]] = counts.get(x["kind"], 0) + 1
MOST_COMMON = max(counts, key=counts.get)
BASELINE = sum(x["kind"] == MOST_COMMON for x in HELD_OUT) / len(HELD_OUT)
for i in (3, 25, 40):
    print(f"note: {NOTES[i]['text']}  ->  {NOTES[i]['kind']}")
print(f"train {len(TRAIN)}, held-out {len(HELD_OUT)}, mean tokens "
      f"{sum(len(x['text'].split()) for x in NOTES) / len(NOTES):.2f}")
print("in the held-out set " + ", ".join(
    f"{t} {sum(x['kind'] == t for x in HELD_OUT) / len(HELD_OUT):.4f}" for t in KINDS))
print(f"most frequent class '{MOST_COMMON}', domain baseline {BASELINE:.4f}")

note: baglanti degistirilmis, kelepce gevsek, dogu hatti  ->  illicit use
note: bodrum katina inilemedi, abone evde degildi, sayac cami kirik, bahce icinde  ->  unreachable
note: vanada sorun yok, gostergede rakam net, sayac durmus, kuzey hatti  ->  ordinary
train 1100, held-out 500, mean tokens 6.68
in the held-out set ordinary 0.4420, unreachable 0.2340, meter fault 0.1920, illicit use 0.1320
most frequent class 'ordinary', domain baseline 0.4420

The second and third example notes show what the fiction makes difficult. The second was generated from the “unreachable” kind but carries a fault record (sayac cami kirik, the meter glass is broken); the third was generated from the “ordinary” kind but the phrase “sayac durmus” (the meter has stopped) comes from the fault phrases. The text does not fully determine the label, it only points to it.

The domain baseline is 0.4420. This number is the rate in the held-out set of a procedure that writes “ordinary” on every note without looking at any word, and every number from here on is read against it.

Three Steps, Twelve Setups

The pipeline is the product of three decisions. Tokenization has three options: coarse (splitting on whitespace only, punctuation stays attached to the word), careful (comma stripped, lowercased), clipped (on top of careful, every word is cut down to its first five characters — the crudest approximation of stemming in an agglutinative language). Weighting has two options: raw frequency, or the product of the frequency with the inverse document frequency. The classifier has two options. The three decisions have twelve combinations, and all twelve are measured on the same held-out set.

# The three steps of the pipeline. Every step is learned from the TRAINING set ONLY.
SPLIT = {"coarse": lambda m: m.split(),
         "careful": lambda m: [w.strip(",").lower() for w in m.split() if w.strip(",")],
         "clipped": lambda m: [w.strip(",").lower()[:5] for w in m.split() if w.strip(",")]}


def build_vocabulary(D, split):
    d = {}
    for x in D:
        for w in set(split(x["text"])):
            d[w] = d.get(w, 0) + 1
    words = sorted(w for w, c in d.items() if c >= 2)
    return ({w: i for i, w in enumerate(words)},
            {i: math.log(len(D) / d[w]) for i, w in enumerate(words)})


def vectorize(text, split, vocab, idf=None):
    v = {}
    for w in split(text):
        if w in vocab:
            v[vocab[w]] = v.get(vocab[w], 0.0) + 1.0
    if idf:
        for i in v:
            v[i] *= idf[i]
    n = math.sqrt(sum(u * u for u in v.values())) or 1.0
    return {i: u / n for i, u in v.items()}


def class_mean(D, V):                       # THE MODEL: the mean vector of every class
    C = [{} for _ in KINDS]
    for x, v in zip(D, V):
        c = C[KIND_INDEX[x["kind"]]]
        for i, u in v.items():
            c[i] = c.get(i, 0.0) + u
    for c in C:
        n = math.sqrt(sum(u * u for u in c.values())) or 1.0
        for i in c:
            c[i] /= n
    return lambda v: max(range(4), key=lambda j: sum(u * C[j].get(i, 0.0)
                                                     for i, u in v.items()))


def linear_model(D, V, size, epochs=150, step=3.0):   # THE MODEL: four-class linear
    W, b = [[0.0] * size for _ in KINDS], [0.0] * 4
    y, n = [KIND_INDEX[x["kind"]] for x in D], len(D)
    for _ in range(epochs):
        gW, gb = [{} for _ in KINDS], [0.0] * 4
        for v, t in zip(V, y):
            o = [sum(W[j][i] * u for i, u in v.items()) + b[j] for j in range(4)]
            e = [math.exp(z - max(o)) for z in o]
            s = sum(e)
            for j in range(4):
                d = e[j] / s - (1.0 if j == t else 0.0)
                gb[j] += d
                for i, u in v.items():
                    gW[j][i] = gW[j].get(i, 0.0) + d * u
        for j in range(4):
            b[j] -= step * gb[j] / n
            for i, g in gW[j].items():
                W[j][i] -= step * g / n
    return lambda v: max(range(4), key=lambda j: sum(W[j][i] * u
                                                     for i, u in v.items()) + b[j])


def score(predict, D, V):
    return sum(predict(v) == KIND_INDEX[x["kind"]] for x, v in zip(D, V)) / len(D)


KEYWORDS = ("muhur", "erisim", "sayac", "gosterge", "kelepce")


def by_hand(text):                          # the hand-counted text measures of the Data Preparation and Feature Engineering course
    w = SPLIT["careful"](text)
    v = {0: len(text) / 60.0, 1: len(w) / 10.0}
    for j, a in enumerate(KEYWORDS):
        v[2 + j] = 1.0 if any(a in u for u in w) else 0.0
    n = math.sqrt(sum(u * u for u in v.values())) or 1.0
    return {i: u / n for i, u in v.items()}


V_TRAIN, V_HELD = [by_hand(x["text"]) for x in TRAIN], [by_hand(x["text"]) for x in HELD_OUT]
BY_HAND_SCORE = score(linear_model(TRAIN, V_TRAIN, 7), HELD_OUT, V_HELD)
GRID = {}
print(f"{'tokenization':<13}{'vocab':>7}{'weighting':>11}{'class mean':>12}{'linear':>10}")
for name, split in SPLIT.items():
    vocab, idf = build_vocabulary(TRAIN, split)
    for weighting, w in (("count", None), ("weighted", idf)):
        V_TRAIN = [vectorize(x["text"], split, vocab, w) for x in TRAIN]
        V_HELD = [vectorize(x["text"], split, vocab, w) for x in HELD_OUT]
        a = score(class_mean(TRAIN, V_TRAIN), HELD_OUT, V_HELD)
        d = score(linear_model(TRAIN, V_TRAIN, len(vocab)), HELD_OUT, V_HELD)
        GRID[(name, weighting)] = (a, d)
        print(f"{name if weighting == 'count' else '':<13}{len(vocab) if weighting == 'count' else '':>7}"
              f"{weighting:>11}{a:>12.4f}{d:>10.4f}")
print(f"baseline {BASELINE:.4f}; seven hand-counted measures + linear model {BY_HAND_SCORE:.4f}")

tokenization   vocab  weighting  class mean    linear
coarse           156      count      0.9040    0.9500
                       weighted      0.9300    0.9580
careful          109      count      0.8940    0.9620
                       weighted      0.9200    0.9580
clipped           89      count      0.8700    0.9500
                       weighted      0.9040    0.9440
baseline 0.4420; seven hand-counted measures + linear model 0.5500

The last line is the second step of the baseline reading of this course. feature-selection/02 of the Data Preparation and Feature Engineering course had extracted five numbers by hand from a field note and measured that they added 0.0317 on the held-out set; the task there was binary suspicion prediction. When the same approach is applied here to the four-class event kind task, it gives 0.5500 — 0.1080 above the baseline. The lowest number the bag of words gives on the same held-out set is 0.8700, the highest 0.9620.

A Step’s Contribution Is Not a Single Number

Twelve setups make it impossible to answer the question “which step adds how many points” with a single subtraction. The contribution of every step depends on the value at which the others stand, and this dependence can be counted: while one step is changed, all the remaining setups are held fixed and the difference is read separately in each of them.

# A step's contribution is not a single number: it is measured separately in every setup of the other steps.
STEPS = [("tokenization coarse -> careful",
          [GRID[("careful", g)][k] - GRID[("coarse", g)][k]
           for g in ("count", "weighted") for k in (0, 1)]),
         ("tokenization careful -> clipped",
          [GRID[("clipped", g)][k] - GRID[("careful", g)][k]
           for g in ("count", "weighted") for k in (0, 1)]),
         ("weighting count -> weighted",
          [GRID[(s, "weighted")][k] - GRID[(s, "count")][k]
           for s in SPLIT for k in (0, 1)]),
         ("classifier mean -> linear",
          [GRID[(s, g)][1] - GRID[(s, g)][0]
           for s in SPLIT for g in ("count", "weighted")]),
         ("representation by hand -> bag of words",
          [GRID[(s, g)][k] - BY_HAND_SCORE for s in SPLIT
           for g in ("count", "weighted") for k in (0, 1)])]
print(f"{'step':<38}{'count':>6}{'lowest':>10}{'highest':>11}{'mean':>10}")
for label, f in STEPS:
    print(f"{label:<38}{len(f):>6}{min(f):>+10.4f}{max(f):>+11.4f}"
          f"{sum(f) / len(f):>+10.4f}")

V, _ = build_vocabulary(TRAIN, SPLIT["careful"])
oov = [sum(w not in V for w in SPLIT["careful"](x["text"])) for x in HELD_OUT]
total = sum(len(SPLIT["careful"](x["text"])) for x in HELD_OUT)
print(f"vocabulary built from training only: {len(V)} words; in the held-out set "
      f"out-of-vocabulary tokens {sum(oov)}/{total}")
best = max(GRID.items(), key=lambda kv: kv[1][1])
print(f"highest setup {best[0][0]} + {best[0][1]} + linear "
      f"{best[1][1]:.4f}; {best[1][1] - BASELINE:+.4f} above the baseline")

step                                   count    lowest    highest      mean
tokenization coarse -> careful             4   -0.0100    +0.0120   -0.0020
tokenization careful -> clipped            4   -0.0240    -0.0120   -0.0165
weighting count -> weighted                6   -0.0060    +0.0340   +0.0140
classifier mean -> linear                  6   +0.0280    +0.0800   +0.0500
representation by hand -> bag of words    12   +0.3200    +0.4120   +0.3787
vocabulary built from training only: 109 words; in the held-out set out-of-vocabulary tokens 0/3305
highest setup careful + count + linear 0.9620; +0.5200 above the baseline

The largest row of the table is at the bottom and it is the real answer of this lesson: almost all of the difference comes from the representation. Replacing the seven hand-counted measures with the bag of words adds at least 0.3200 in all twelve setups. Changing the classifier adds 0.0500 on average, weighting 0.0140. The difference between seven numbers and one hundred and nine numbers is seven and a half times the difference between the class mean and the linear model.

The weighting row changes sign and this is the second finding of this lesson. When working with the class mean, the inverse document frequency gives a positive contribution in all three tokenizations, the highest being 0.0340; when working with the linear model it drops to the negative side in two setups. The reason lies in the structure of the procedures: the class mean trusts all words equally, so suppressing the high-frequency stop words helps it. The linear model has already learned that suppression through its weights, and a second suppression coming from outside adds no information. The question “how many points does term weighting add” has no single answer; it is a question asked before the classifier was written.

Both tokenization rows are negative. The move from coarse splitting to careful brings the vocabulary down from 156 to 109 and costs 0.0020 on average; suffix clipping brings the vocabulary down to 89 and costs 0.0165. Clipping gathers the forms sayac, sayaci, sayacin (meter, the meter, of the meter) into a single token — but it also erases the distinction between “kapak temiz” (cover clean) and “kapakta iz yok” (no mark on the cover), because their distinguishing words intersect at five characters. Shrinking the vocabulary is a measure of cost, not a measure of performance; the two do not have to move in the same direction.

The last two lines show the cost of the pipeline rule in this set. Because the vocabulary is built from the training set only, some words of the held-out set would be expected to remain outside; here the number of out-of-vocabulary tokens is zero out of 3,305, because the vocabulary of the fiction is closed and 1,100 notes are enough to see all the phrases. The rule has no cost here, but it was still applied: that the cost is zero is known only after it has been measured.

Summary

• In document-level classification the decision unit is the document; the number of tokens of the input is variable, and the output is a single label.
• The domain baseline is the most frequent class and gives 0.4420; seven hand-counted text measures give 0.5500, the bag of words with a linear model gives 0.9620.
• Almost all of the difference comes from the representation: the move from hand counting to the bag of words adds at least 0.3200 in all twelve setups, the classifier step adds 0.0500 on average.
• A step’s contribution is not a single number: term weighting rises to 0.0340 with the class mean and drops to minus 0.0060 with the linear model.
• Shrinking the vocabulary does not raise performance: suffix clipping brings the vocabulary from 109 down to 89 and costs 0.0165 on average.

Next Step

In this lesson the bag of words passed the baseline by 0.5200, but 1,100 labeled notes were spent for it. Every note is the counterpart of a field visit, and there can be a hundred of them at hand. Unlabeled notes, on the other hand, are plentiful: the measurement network produces thousands of notes every period and none of them has an event kind written on it. A proxy task can be built from these notes — a word is hidden and predicted from its neighbors — and the representation learned there can be carried over to the labeled task. The next lesson measures this loop: how far an embedding pretrained with masked prediction passes a setup trained from scratch as the number of labeled examples shrinks, and at which number of examples the difference closes.