Text Preprocessing

The Deep Learning course built the architecture over twenty-four lessons, but the input was always a ready-made number: a column, a grid cell, a reading sequence, an event code. With a text in hand, the situation is different. In a note of a few words that the field crew files on a subscriber record, there is not a single number the model can read; what produces the number is a sequence of decisions.

The measure of this course is set from here. Every lesson puts three numbers side by side: the domain baseline, the number of the most naive procedure on the same metric for that task; the number of the method; and the difference, written together with whether the difference came from the representation, the model or the metric. The rule is this: a domain metric cannot be read without the number of that domain’s most naive procedure on the same metric. A number whose domain baseline is not written is a number that has not been measured. Here the baseline is splitting the text on whitespace, there is no model, and whatever the difference turns out to be comes from the representation alone.

• NL1. The notes are fictional: they are generated within the lesson, seed 20260218, and none of them comes from a source system. The set is 1,800 notes and is the expanded version of the field-note column of the Data Preparation and Feature Engineering course.
• NL2. The vocabulary is built from 85 roots: 24 inflected nouns, 10 verbs, 11 adjectives and 40 rare nouns. The weights of the rare nouns drop quickly; a few appear in the notes only once or twice.
• NL3. The labels are fictional: the event kind (ordinary, fault, unreachable, suspected illicit use) and the binary illicit-use suspicion. The probability of suspicion depends only on the event kind.
• NL4. A token is the smallest piece into which a text is split. In the JavaScript and TypeScript course, the unit produced by the language parser was called a lexical token; the distinction is kept, because there the split is exact by grammar and here it is a decision.
• NL5. The first 1,200 notes are the training set and the remaining 600 notes are the held-out set. The vocabulary is built from the training set only; a token that is seen in the held-out set and is not in the vocabulary is out-of-vocabulary.
• NL6. Text normalization and string procedures were measured in the data cleaning part of the Data Collection and Preparation course; the procedure is not repeated, only the number that the decision moves is measured.
• NL7. The metric is vocabulary size and out-of-vocabulary rate (smaller is better) and it is the same on every row.
• NL8. The accuracy of the stemmer is measured against the root mapping of the setup; the lemmatization table is only the part of the same mapping that was seen in the training set.

The Setup of the Notes

A note consists of one or a few short phrases: the first phrase carries the event kind, and the extra phrase and the rare-noun phrase are independent of the kind and are noise.

# COMMON — FICTIONAL. The field notes are generated within the lesson, seed 20260218;
# they do not come from a source system. The vocabulary has 45 inflected roots and
# 40 rare nouns.
SEED, NOTE_COUNT, M32 = 20260218, 1800, 0xFFFFFFFF

NOUNS = {  # 0 base, 1 genitive, 2 locative, 3 possessive, 4 plural
    "sayac": "sayac sayacin sayacta sayaci sayaclar",
    "kapak": "kapak kapagin kapakta kapagi kapaklar",
    "muhur": "muhur muhurun muhurde muhuru muhurler",
    "vana": "vana vananin vanada vanasi vanalar",
    "kelepce": "kelepce kelepcenin kelepcede kelepcesi kelepceler",
    "boru": "boru borunun boruda borusu borular",
    "pano": "pano panonun panoda panosu panolar",
    "kapi": "kapi kapinin kapida kapisi kapilar",
    "zil": "zil zilin zilde zili ziller",
    "bahce": "bahce bahcenin bahcede bahcesi bahceler",
    "apartman": "apartman apartmanin apartmanda apartmani apartmanlar",
    "bodrum": "bodrum bodrumun bodrumda bodrumu bodrumlar",
    "giris": "giris girisin giriste girisi girisler",
    "adres": "adres adresin adreste adresi adresler",
    "okuma": "okuma okumanin okumada okumasi okumalar",
    "olcum": "olcum olcumun olcumde olcumu olcumler",
    "erisim": "erisim erisimin erisimde erisimi erisimler",
    "ziyaret": "ziyaret ziyaretin ziyarette ziyareti ziyaretler",
    "abone": "abone abonenin abonede abonesi aboneler",
    "arac": "arac aracin aracta araci araclar",
    "kir": "kir kirin kirde kiri kirler",
    "iz": "iz izin izde izi izler",
    "sokak": "sokak sokagin sokakta sokagi sokaklar",
    "ev": "ev evin evde evi evler",
}
ADJECTIVES = "kirik gevsek kirli normal bozuk acik kapali eksik temiz yeni yok".split()
VERBS = {  # 0 passive past, 1 negative
    "calis": "calisiyor calismiyor", "don": "donuyor donmuyor", "oku": "okundu okunmadi",
    "ac": "acildi acilmadi", "gir": "girildi girilemedi", "ulas": "ulasildi ulasilamadi",
    "tekrarla": "tekrarlandi tekrarlanmadi", "bul": "bulundu bulunmadi",
    "sagla": "saglandi saglanamadi", "degil": "degil degildi",
}
RARE_NOUNS = """kablo kablosu kabloda|cam cami camda|tel teli telde|mandal mandali mandalda
merdiven merdiveni merdivende|tahta tahtasi tahtada|cukur cukuru cukurda|kar kari karda
buz buzu buzda|cop copu copta|duvar duvari duvarda|kutu kutusu kutuda|vida vidasi vidada
conta contasi contada|filtre filtresi filtrede|bant banti bantta|etiket etiketi etikette
kilit kiliti kilitte|asansor asansoru asansorde|zemin zemini zeminde|tavan tavani tavanda
kuyu kuyusu kuyuda|depo deposu depoda|ocak ocagi ocakta|direk diregi direkte
kanal kanali kanalda|numara numarasi numarada|cati catisi catida|baca bacasi bacada
kova kovasi kovada|levha levhasi levhada|somun somunu somunda|tapa tapasi tapada
izolasyon izolasyonu izolasyonda|supap supabi supapta|menfez menfezi menfezde
kayit kaydi kayitta|temel temeli temelde|kose kosesi kosede|koruma korumasi korumada"""
PHRASES = {
    "ordinary": ["{sayac.0} {calis.0}", "{okuma.0} normal", "{olcum.0} {tekrarla.0}",
               "{sayac.1} {erisim.3} {sagla.0}", "{kapak.0} temiz", "{vana.0} acik",
               "{sayac.2} {iz.0} {bul.1}", "{olcum.0} icin {erisim.0} {sagla.0}",
               "{sayac.1} {kapak.3} yeni", "{muhur.0} ile {kapak.0} temiz"],
    "fault": ["{sayac.0} {don.1}", "{kapak.0} kirik", "{pano.0} bozuk",
              "{sayac.1} {kapak.3} kirik", "{olcum.0} {oku.1}", "{vana.0} bozuk",
              "{sayac.2} {kir.0} {bul.0}", "{sayac.4} {calis.1}",
              "{sayac.0} {calis.0} ama {okuma.0} yok",
              "{pano.1} {kapak.3} eksik"],
    "unreachable": ["{kapi.0} {ac.1}", "{zil.0} {calis.1}", "{abone.0} {ev.2} {degil.1}",
                    "{erisim.0} {sagla.1}", "{bodrum.0} {kapi.3} kapali",
                    "{apartman.1} {giris.3} kapali", "{adres.2} {ulas.1}",
                    "{bahce.1} {kapi.3} kapali", "{apartman.2} {zil.0} {calis.1}",
                    "{sayac.2} {erisim.0} yok"],
    "illicit": ["{muhur.0} kirik", "{kelepce.0} gevsek", "{muhur.2} {iz.0} {bul.0}",
              "{sayac.1} {muhur.3} kirik", "{kelepce.2} {iz.0} {bul.0}",
              "{muhur.0} eksik", "{vana.1} {kelepce.3} gevsek",
              "{muhur.1} {iz.3} {bul.0}", "{boru.2} {kelepce.0} eksik",
              "{kapak.2} bir {iz.0} {bul.0}"],
}
EXTRA_PHRASES = ["{ziyaret.0} {tekrarla.0}", "{sokak.2} {arac.0} {bul.0}", "{adres.0} yeni",
           "{bahce.1} {kapi.3} acik", "{apartman.2} {gir.0}", "{ev.1} {kapi.3} acik",
           "{kapak.0} kirli", "{olcum.0} {tekrarla.1}", "{arac.0} {sokak.2}",
           "bu {ziyaret.2} {olcum.0} {tekrarla.0}"]
KINDS = [("ordinary", 0.36, 0.04), ("fault", 0.26, 0.28),
         ("unreachable", 0.16, 0.15), ("illicit", 0.22, 0.82)]
FORMS = {k: v.split() for d in (NOUNS, VERBS) for k, v in d.items()}
RARE = [s.split() for s in RARE_NOUNS.replace("\n", "|").split("|")]
WEIGHTS = [1 / (i + 1) ** 1.6 for i in range(len(RARE))]
WEIGHTS = [w / sum(WEIGHTS) for w in WEIGHTS]
ROOT = {f: k for k, fs in FORMS.items() for f in fs}
ROOT.update({f: fs[0] for fs in RARE for f in fs})
ROOT.update({w: w for w in ADJECTIVES})
STOP_WORDS = ["ve", "ile", "bir", "bu", "ama", "icin"]
ROOT.update({w: w for w in STOP_WORDS})


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def render_phrase(template):
    parts = []
    for part in template.split():
        if part.startswith("{"):
            name, _, i = part[1:-1].partition(".")
            parts.append(FORMS[name][int(i)])
        else:
            parts.append(part)
    return " ".join(parts)


DATA = []
for i in range(NOTE_COUNT):
    r = rng(SEED + i)
    kind = KINDS[pick(r(), [x[1] for x in KINDS])]
    clauses = [render_phrase(PHRASES[kind[0]][int(r() * 10)])]
    if r() < 0.55:
        clauses.append(render_phrase(EXTRA_PHRASES[int(r() * 10)]))
    if r() < 0.30:
        clauses.append(render_phrase(PHRASES[kind[0]][int(r() * 10)]))
    if r() < 0.22:
        forms, adjective = RARE[pick(r(), WEIGHTS)], ADJECTIVES[int(r() * len(ADJECTIVES))]
        clauses.append(forms[2] + " kir bulundu" if r() < 0.34
                       else forms[int(r() * 2)] + " " + adjective)
    note = clauses[0]
    for x in clauses[1:]:
        note += (" ve " if r() < 0.22 else ", ") + x
    if r() < 0.52:
        note += ", okuma " + str(240 + int(r() * 1760))
    DATA.append({"id": 10001 + i, "note": note[0].upper() + note[1:] + ".",
                 "kind": kind[0], "suspicious": int(r() < kind[2])})

TRAIN, HELD_OUT = DATA[:1200], DATA[1200:]

for i in (0, 3, 41):
    print(f"{DATA[i]['kind']:<12} {DATA[i]['suspicious']}  {DATA[i]['note']}")
print(f"\n{len(DATA)} notes, {len(ROOT)} vocabulary forms, "
      f"{len(FORMS) + len(RARE) + len(ADJECTIVES)} roots")
for name, _, _ in KINDS:
    n = [x for x in DATA if x["kind"] == name]
    print(f"{name:<12} {len(n):>5} notes   suspicion rate "
          f"{sum(x['suspicious'] for x in n) / len(n):.4f}")
print(f"{'total':<12} {len(DATA):>5} notes   suspicion rate "
      f"{sum(x['suspicious'] for x in DATA) / len(DATA):.4f}")

illicit      1  Vananin kelepcesi gevsek.
ordinary     0  Olcum icin erisim saglandi, sayacin kapagi yeni, mandalda kir bulundu.
illicit      1  Boruda kelepce eksik, okuma 1505.

1800 notes, 277 vocabulary forms, 85 roots
ordinary       638 notes   suspicion rate 0.0486
fault          446 notes   suspicion rate 0.3072
unreachable    305 notes   suspicion rate 0.1377
illicit        411 notes   suspicion rate 0.7835
total         1800 notes   suspicion rate 0.2956

The suspicion rate varies between 0.0486 and 0.7835 by kind: the text is related to the label but does not determine it, and 89 of the 411 notes that carry an illicit phrase have no suspicion. This is the ceiling for the later lessons, and the most frequent class holds a share of 0.7044.

Three Tokenization Decisions

The most naive procedure is to split on whitespace: punctuation stays attached to the word, and a capital letter at the start of a sentence counts as a separate type. The second decision drops the punctuation and lowercases. The third decision adds rule-based stemming on top: the longest suffix matching at the end is removed, provided that at least three characters remain.

SUFFIXES = ["lari", "leri", "lar", "ler", "nin", "nun", "dan", "den", "tan", "ten",
            "dir", "tir", "dur", "da", "de", "ta", "te", "in", "un", "yi", "yu",
            "si", "su", "li", "lu", "di", "du", "ti", "tu", "i", "u", "a", "e"]


def decision_1(note):  # whitespace
    return note.split()


def decision_2(note):  # punctuation dropped, lowercased
    return [t for t in (s.strip(",.").lower() for s in note.split()) if t]


def stem(token):
    for suffix in SUFFIXES:
        if token.endswith(suffix) and len(token) - len(suffix) >= 3:
            return token[:-len(suffix)]
    return token


def decision_3(note):  # + stemming
    return [stem(t) for t in decision_2(note)]


def is_number(token):
    return token.strip(",.").isdigit()


def measure(f, name, number_share=True):
    train_tokens = [t for x in TRAIN for t in f(x["note"])]
    held_tokens = [t for x in HELD_OUT for t in f(x["note"])]
    vocabulary = set(train_tokens)
    out_of_vocab = [t for t in held_tokens if t not in vocabulary]
    extra = (f"   number share {sum(map(is_number, out_of_vocab)) / len(out_of_vocab):.4f}"
             if number_share else "")
    print(f"{name:<28} {len(train_tokens) + len(held_tokens):>8} {len(vocabulary):>10}"
          f" {len(out_of_vocab):>17} {len(out_of_vocab) / len(held_tokens):>8.4f}{extra}")


print(f"{'tokenization decision':<28} {'tokens':>8} {'vocabulary':>10}"
      f" {'out-of-vocabulary':>17} {'rate':>8}")
measure(decision_1, "whitespace")
measure(decision_2, "drop punctuation, lowercase")
measure(decision_3, "+ stemming")


def fold(f):
    return lambda note: ["<number>" if is_number(t) else t for t in f(note)]


print()
measure(fold(decision_1), "whitespace", False)
measure(fold(decision_2), "drop punctuation, lowercase", False)
measure(fold(decision_3), "+ stemming", False)

tokenization decision          tokens vocabulary out-of-vocabulary     rate
whitespace                      12044        748               225   0.0565   number share 0.9467
drop punctuation, lowercase     12044        669               225   0.0565   number share 0.9467
+ stemming                      12044        641               222   0.0557   number share 0.9595

whitespace                      12044        203                12   0.0030
drop punctuation, lowercase     12044        124                12   0.0030
+ stemming                      12044         96                 9   0.0023

The three decisions produce the same 12,044 tokens but three different vocabularies: 748, 669, 641. The baseline vocabulary is inflated by 107 tokens, because kapali (closed) and kapali, and Kapi (door) and kapi are counted as distinct types. The out-of-vocabulary rate, on the other hand, is the same in the first two decisions: 0.0565. Vocabulary size and out-of-vocabulary rate do not move in the same direction, and the reason is written in the last column: 0.9467 of the out-of-vocabulary tokens are numbers. The reading value is different in every note, and a number that was not seen in training falls out of the vocabulary in the held-out set.

When numbers are folded into a single token, the table reads differently: the vocabulary is 203, 124 and 96, the rate 0.0030 and 0.0023. That folding brings the vocabulary down to only 203 under the whitespace decision is a dependence between decisions: because the punctuation stays attached, 1107. is not counted as a number.

The triple can now be written: baseline 748 and 0.0565, all three decisions together 641 and 0.0557. The difference is 107 tokens in the vocabulary and 0.0008 in the out-of-vocabulary rate, and it comes entirely from the representation. In the number-folded table, stemming brings the out-of-vocabulary tokens down from 12 to 9; a difference of three tokens read over 3,985 tokens says nothing on its own.

How Many Stems One Root Has

Turkish is an agglutinative language: a root yields dozens of surface forms through successive suffixes. The aim of the stemmer is to collect them in a single stem; since the root mapping is at hand, how close it gets to the aim can be counted.

surface_forms = sorted({t for x in DATA for t in decision_2(x["note"]) if t in ROOT})
root_stems, stem_roots = {}, {}
for t in surface_forms:
    root_stems.setdefault(ROOT[t], set()).add(stem(t))
    stem_roots.setdefault(stem(t), set()).add(ROOT[t])
scattered = sorted(k for k, v in root_stems.items() if len(v) > 1)
merging = sorted(g for g, v in stem_roots.items() if len(v) > 1)
print(f"surface forms occurring in the notes {len(surface_forms)}, true roots {len(root_stems)},"
      f" stems produced by the stemmer {len(stem_roots)}")
print(f"roots scattered over several stems {len(scattered)}, stems merging several roots"
      f" {len(merging)}")
for k in scattered[:8]:
    print(f"  {k:<10} -> {' '.join(sorted(root_stems[k]))}")
for g in merging:
    print(f"  {g:<10} <- {' '.join(sorted(stem_roots[g]))}")

# The lemmatization table recognizes only the forms seen in the training set.
TABLE = {t: ROOT[t] for x in TRAIN for t in decision_2(x["note"]) if t in ROOT}
held_tokens = [t for x in HELD_OUT for t in decision_2(x["note"]) if not is_number(t)]
print(f"\ntable recognizes {len(TABLE)} forms; coverage on non-number held-out tokens"
      f" {sum(t in TABLE for t in held_tokens) / len(held_tokens):.4f}")
measure(fold(lambda note: [TABLE.get(t, t) for t in decision_2(note)]),
        "lemmatization (table)", False)
measure(fold(decision_3), "stemming (rules)", False)

surface forms occurring in the notes 134, true roots 85, stems produced by the stemmer 103
roots scattered over several stems 19, stems merging several roots 1
  apartman   -> apartma apartman
  baca       -> bac baca
  bant       -> ban bant
  bul        -> bulun bulunma
  calis      -> calisiyor calismiyor
  ev         -> evd evin
  iz         -> iz izi
  kapak      -> kapag kapak
  kir        <- kir kirli

table recognizes 123 forms; coverage on non-number held-out tokens 0.9967
lemmatization (table)           12044         82                11   0.0028
stemming (rules)                12044         96                 9   0.0023

134 surface forms occur in the notes, they belong to 85 roots, and the stemmer produces 103 stems: 18 stems away from the aim. The flaw runs in two directions. Under-stemming is seen in 19 of the 85 roots and comes from four causes: consonant softening (kapak (cover) and kapagi (its cover)), the longest match picking the wrong suffix (when nin is removed from apartmanin (of the apartment building), apartma is left), the last letter of the root being mistaken for a suffix (baca (chimney) and bacada (in the chimney)), and the condition of at least three characters (izi (its mark)). Verbs are never resolved, because the suffix list was written for noun inflection. Over-stemming happens once: removing li from kirli (dirty) merges it with kir (dirt), which is a separate root; kapak kirli (cover dirty) and sayacta kir bulundu (dirt was found in the meter) share the same stem.

Lemmatization offers a different balance. The table recognizes the 123 forms seen in training and brings the vocabulary down to 82 tokens; this is smaller than the 96 of the rule and closer to the true number of roots. Its cost is in the last two rows: the coverage is 0.9967, a form that is not recognized passes through as it is, and the out-of-vocabulary tokens stay at 11, while the rule brings them down to 9. One is more accurate where it recognizes, the other works even where it does not.

A detail lies under the measurement: the form sayaci (the meter; its meter) can be produced with both the accusative and the possessive suffix. No tokenization decision can tell them apart, because the decision looks only at characters.

Summary

• The notes are fictional and generated within the lesson: 1,800 notes, 85 roots, seed 20260218; the suspicion rate varies between 0.0486 and 0.7835 by kind, and the most frequent class holds a share of 0.7044.
• Three tokenization decisions yield three different vocabularies from the same 12,044 tokens: 748, 669, 641. The domain baseline is splitting on whitespace, and it inflates the vocabulary by 107 tokens.
• The out-of-vocabulary rate is the same in the first two decisions (0.0565), because 0.9467 of the out-of-vocabulary tokens are numbers; when numbers are folded, the vocabulary becomes 203, 124, 96 and the rate falls to 0.0030 and 0.0023. The difference comes entirely from the representation: there is no model, and the metric is the same on every row.
• The stemmer splits 85 roots into 103 stems; 19 roots scatter over several stems, and one stem merges two roots. Table-based lemmatization brings the vocabulary down to 82 but has a coverage of 0.9967 and leaves the out-of-vocabulary tokens at 11.

Next Step

The text has been split into tokens, the forms have been reduced to stems and roots, and two metrics have been counted. What came out is still a list of words; not a single number that a model will see has been produced. The Data Preparation and Feature Engineering course had extracted five numbers by hand from the same note field and measured a contribution of 0.0317 on the held-out set; those five numbers are the baseline of this course. The next lesson turns the whole vocabulary into numbers: every token becomes a column, and term weighting is placed on top. The measure is how far above that line it goes.