--- title: 'Text Preprocessing' source: 'https://academia.sh/en/courses/nlp-and-vision/text-preprocessing' course: 'Natural Language Processing and Computer Vision' language: en updated: '2026-10-21T05:03:27+00:00' license: 'CC BY-SA 4.0' objectives: - 'Measures that the same text yields three different vocabularies under three tokenization decisions and shows that vocabulary size and out-of-vocabulary rate do not move in the same direction' - 'Compares rule-based stemming and table-based lemmatization by vocabulary size and out-of-vocabulary rate on the same held-out set' - 'Explains why the agglutinative structure of Turkish makes stemming hard: how many roots scatter over several stems and how many stems collect several roots' --- # Text Preprocessing Measuring how much the decision moves the number when short Turkish field notes are split into tokens: the domain baseline of splitting on whitespace gives a vocabulary of 748 tokens and an out-of-vocabulary rate of 0.0565, while dropping punctuation and lowercasing brings the vocabulary to 669 and adding stemming brings it to 641 and the rate to 0.0557. 0.9467 of the out-of-vocabulary tokens are numbers; when numbers are folded into a single token the vocabulary becomes 203, 124 and 96. The rule-based stemmer splits 85 roots into 103 stems, scatters 19 roots over several stems and merges two roots in one stem; table-based lemmatization cuts the vocabulary to 82 but has a coverage of 0.9967 and can do nothing with a form it does not recognize. The Deep Learning course built the architecture over twenty-four lessons, but the input was always a ready-made number: a column, a grid cell, a reading sequence, an event code. With a text in hand, the situation is different. In a note of a few words that the field crew files on a subscriber record, there is not a single number the model can read; what produces the number is a sequence of decisions. The measure of this course is set from here. Every lesson puts three numbers side by side: the **domain baseline**, the number of the most naive procedure on the same metric for that task; the **number of the method**; and the **difference**, written together with whether the difference came from the representation, the model or the metric. The rule is this: a domain metric cannot be read without the number of that domain's most naive procedure on the same metric. **A number whose domain baseline is not written is a number that has not been measured.** Here the baseline is splitting the text on whitespace, there is no model, and whatever the difference turns out to be comes from the representation alone. - **NL1.** The notes are **fictional**: they are generated within the lesson, seed 20260218, and none of them comes from a source system. The set is 1,800 notes and is the expanded version of the field-note column of the Data Preparation and Feature Engineering course. - **NL2.** The vocabulary is built from 85 roots: 24 inflected nouns, 10 verbs, 11 adjectives and 40 rare nouns. The weights of the rare nouns drop quickly; a few appear in the notes only once or twice. - **NL3.** The labels are **fictional**: the event kind (ordinary, fault, unreachable, suspected illicit use) and the binary illicit-use suspicion. The probability of suspicion depends only on the event kind. - **NL4.** A **token** is the smallest piece into which a text is split. In the JavaScript and TypeScript course, the unit produced by the language parser was called a **lexical token**; the distinction is kept, because there the split is exact by grammar and here it is a decision. - **NL5.** The first 1,200 notes are the training set and the remaining 600 notes are the held-out set. The vocabulary is built **from the training set only**; a token that is seen in the held-out set and is not in the vocabulary is **out-of-vocabulary**. - **NL6.** Text normalization and string procedures were measured in the data cleaning part of the Data Collection and Preparation course; the procedure is not repeated, only the number that the decision moves is measured. - **NL7.** The metric is vocabulary size and out-of-vocabulary rate (smaller is better) and it is the same on every row. - **NL8.** The accuracy of the stemmer is measured against the root mapping of the setup; the lemmatization table is **only the part of the same mapping that was seen in the training set**. ## The Setup of the Notes A note consists of one or a few short phrases: the first phrase carries the event kind, and the extra phrase and the rare-noun phrase are independent of the kind and are noise. ```python # COMMON — FICTIONAL. The field notes are generated within the lesson, seed 20260218; # they do not come from a source system. The vocabulary has 45 inflected roots and # 40 rare nouns. SEED, NOTE_COUNT, M32 = 20260218, 1800, 0xFFFFFFFF NOUNS = { # 0 base, 1 genitive, 2 locative, 3 possessive, 4 plural "sayac": "sayac sayacin sayacta sayaci sayaclar", "kapak": "kapak kapagin kapakta kapagi kapaklar", "muhur": "muhur muhurun muhurde muhuru muhurler", "vana": "vana vananin vanada vanasi vanalar", "kelepce": "kelepce kelepcenin kelepcede kelepcesi kelepceler", "boru": "boru borunun boruda borusu borular", "pano": "pano panonun panoda panosu panolar", "kapi": "kapi kapinin kapida kapisi kapilar", "zil": "zil zilin zilde zili ziller", "bahce": "bahce bahcenin bahcede bahcesi bahceler", "apartman": "apartman apartmanin apartmanda apartmani apartmanlar", "bodrum": "bodrum bodrumun bodrumda bodrumu bodrumlar", "giris": "giris girisin giriste girisi girisler", "adres": "adres adresin adreste adresi adresler", "okuma": "okuma okumanin okumada okumasi okumalar", "olcum": "olcum olcumun olcumde olcumu olcumler", "erisim": "erisim erisimin erisimde erisimi erisimler", "ziyaret": "ziyaret ziyaretin ziyarette ziyareti ziyaretler", "abone": "abone abonenin abonede abonesi aboneler", "arac": "arac aracin aracta araci araclar", "kir": "kir kirin kirde kiri kirler", "iz": "iz izin izde izi izler", "sokak": "sokak sokagin sokakta sokagi sokaklar", "ev": "ev evin evde evi evler", } ADJECTIVES = "kirik gevsek kirli normal bozuk acik kapali eksik temiz yeni yok".split() VERBS = { # 0 passive past, 1 negative "calis": "calisiyor calismiyor", "don": "donuyor donmuyor", "oku": "okundu okunmadi", "ac": "acildi acilmadi", "gir": "girildi girilemedi", "ulas": "ulasildi ulasilamadi", "tekrarla": "tekrarlandi tekrarlanmadi", "bul": "bulundu bulunmadi", "sagla": "saglandi saglanamadi", "degil": "degil degildi", } RARE_NOUNS = """kablo kablosu kabloda|cam cami camda|tel teli telde|mandal mandali mandalda merdiven merdiveni merdivende|tahta tahtasi tahtada|cukur cukuru cukurda|kar kari karda buz buzu buzda|cop copu copta|duvar duvari duvarda|kutu kutusu kutuda|vida vidasi vidada conta contasi contada|filtre filtresi filtrede|bant banti bantta|etiket etiketi etikette kilit kiliti kilitte|asansor asansoru asansorde|zemin zemini zeminde|tavan tavani tavanda kuyu kuyusu kuyuda|depo deposu depoda|ocak ocagi ocakta|direk diregi direkte kanal kanali kanalda|numara numarasi numarada|cati catisi catida|baca bacasi bacada kova kovasi kovada|levha levhasi levhada|somun somunu somunda|tapa tapasi tapada izolasyon izolasyonu izolasyonda|supap supabi supapta|menfez menfezi menfezde kayit kaydi kayitta|temel temeli temelde|kose kosesi kosede|koruma korumasi korumada""" PHRASES = { "ordinary": ["{sayac.0} {calis.0}", "{okuma.0} normal", "{olcum.0} {tekrarla.0}", "{sayac.1} {erisim.3} {sagla.0}", "{kapak.0} temiz", "{vana.0} acik", "{sayac.2} {iz.0} {bul.1}", "{olcum.0} icin {erisim.0} {sagla.0}", "{sayac.1} {kapak.3} yeni", "{muhur.0} ile {kapak.0} temiz"], "fault": ["{sayac.0} {don.1}", "{kapak.0} kirik", "{pano.0} bozuk", "{sayac.1} {kapak.3} kirik", "{olcum.0} {oku.1}", "{vana.0} bozuk", "{sayac.2} {kir.0} {bul.0}", "{sayac.4} {calis.1}", "{sayac.0} {calis.0} ama {okuma.0} yok", "{pano.1} {kapak.3} eksik"], "unreachable": ["{kapi.0} {ac.1}", "{zil.0} {calis.1}", "{abone.0} {ev.2} {degil.1}", "{erisim.0} {sagla.1}", "{bodrum.0} {kapi.3} kapali", "{apartman.1} {giris.3} kapali", "{adres.2} {ulas.1}", "{bahce.1} {kapi.3} kapali", "{apartman.2} {zil.0} {calis.1}", "{sayac.2} {erisim.0} yok"], "illicit": ["{muhur.0} kirik", "{kelepce.0} gevsek", "{muhur.2} {iz.0} {bul.0}", "{sayac.1} {muhur.3} kirik", "{kelepce.2} {iz.0} {bul.0}", "{muhur.0} eksik", "{vana.1} {kelepce.3} gevsek", "{muhur.1} {iz.3} {bul.0}", "{boru.2} {kelepce.0} eksik", "{kapak.2} bir {iz.0} {bul.0}"], } EXTRA_PHRASES = ["{ziyaret.0} {tekrarla.0}", "{sokak.2} {arac.0} {bul.0}", "{adres.0} yeni", "{bahce.1} {kapi.3} acik", "{apartman.2} {gir.0}", "{ev.1} {kapi.3} acik", "{kapak.0} kirli", "{olcum.0} {tekrarla.1}", "{arac.0} {sokak.2}", "bu {ziyaret.2} {olcum.0} {tekrarla.0}"] KINDS = [("ordinary", 0.36, 0.04), ("fault", 0.26, 0.28), ("unreachable", 0.16, 0.15), ("illicit", 0.22, 0.82)] FORMS = {k: v.split() for d in (NOUNS, VERBS) for k, v in d.items()} RARE = [s.split() for s in RARE_NOUNS.replace("\n", "|").split("|")] WEIGHTS = [1 / (i + 1) ** 1.6 for i in range(len(RARE))] WEIGHTS = [w / sum(WEIGHTS) for w in WEIGHTS] ROOT = {f: k for k, fs in FORMS.items() for f in fs} ROOT.update({f: fs[0] for fs in RARE for f in fs}) ROOT.update({w: w for w in ADJECTIVES}) STOP_WORDS = ["ve", "ile", "bir", "bu", "ama", "icin"] ROOT.update({w: w for w in STOP_WORDS}) def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def render_phrase(template): parts = [] for part in template.split(): if part.startswith("{"): name, _, i = part[1:-1].partition(".") parts.append(FORMS[name][int(i)]) else: parts.append(part) return " ".join(parts) DATA = [] for i in range(NOTE_COUNT): r = rng(SEED + i) kind = KINDS[pick(r(), [x[1] for x in KINDS])] clauses = [render_phrase(PHRASES[kind[0]][int(r() * 10)])] if r() < 0.55: clauses.append(render_phrase(EXTRA_PHRASES[int(r() * 10)])) if r() < 0.30: clauses.append(render_phrase(PHRASES[kind[0]][int(r() * 10)])) if r() < 0.22: forms, adjective = RARE[pick(r(), WEIGHTS)], ADJECTIVES[int(r() * len(ADJECTIVES))] clauses.append(forms[2] + " kir bulundu" if r() < 0.34 else forms[int(r() * 2)] + " " + adjective) note = clauses[0] for x in clauses[1:]: note += (" ve " if r() < 0.22 else ", ") + x if r() < 0.52: note += ", okuma " + str(240 + int(r() * 1760)) DATA.append({"id": 10001 + i, "note": note[0].upper() + note[1:] + ".", "kind": kind[0], "suspicious": int(r() < kind[2])}) TRAIN, HELD_OUT = DATA[:1200], DATA[1200:] for i in (0, 3, 41): print(f"{DATA[i]['kind']:<12} {DATA[i]['suspicious']} {DATA[i]['note']}") print(f"\n{len(DATA)} notes, {len(ROOT)} vocabulary forms, " f"{len(FORMS) + len(RARE) + len(ADJECTIVES)} roots") for name, _, _ in KINDS: n = [x for x in DATA if x["kind"] == name] print(f"{name:<12} {len(n):>5} notes suspicion rate " f"{sum(x['suspicious'] for x in n) / len(n):.4f}") print(f"{'total':<12} {len(DATA):>5} notes suspicion rate " f"{sum(x['suspicious'] for x in DATA) / len(DATA):.4f}") ``` ``` illicit 1 Vananin kelepcesi gevsek. ordinary 0 Olcum icin erisim saglandi, sayacin kapagi yeni, mandalda kir bulundu. illicit 1 Boruda kelepce eksik, okuma 1505. 1800 notes, 277 vocabulary forms, 85 roots ordinary 638 notes suspicion rate 0.0486 fault 446 notes suspicion rate 0.3072 unreachable 305 notes suspicion rate 0.1377 illicit 411 notes suspicion rate 0.7835 total 1800 notes suspicion rate 0.2956 ``` The suspicion rate varies between 0.0486 and 0.7835 by kind: the text is related to the label but does not determine it, and 89 of the 411 notes that carry an illicit phrase have no suspicion. This is the ceiling for the later lessons, and the most frequent class holds a share of 0.7044. ## Three Tokenization Decisions The most naive procedure is to split on whitespace: punctuation stays attached to the word, and a capital letter at the start of a sentence counts as a separate type. The second decision drops the punctuation and lowercases. The third decision adds rule-based **stemming** on top: the longest suffix matching at the end is removed, provided that at least three characters remain. ```python SUFFIXES = ["lari", "leri", "lar", "ler", "nin", "nun", "dan", "den", "tan", "ten", "dir", "tir", "dur", "da", "de", "ta", "te", "in", "un", "yi", "yu", "si", "su", "li", "lu", "di", "du", "ti", "tu", "i", "u", "a", "e"] def decision_1(note): # whitespace return note.split() def decision_2(note): # punctuation dropped, lowercased return [t for t in (s.strip(",.").lower() for s in note.split()) if t] def stem(token): for suffix in SUFFIXES: if token.endswith(suffix) and len(token) - len(suffix) >= 3: return token[:-len(suffix)] return token def decision_3(note): # + stemming return [stem(t) for t in decision_2(note)] def is_number(token): return token.strip(",.").isdigit() def measure(f, name, number_share=True): train_tokens = [t for x in TRAIN for t in f(x["note"])] held_tokens = [t for x in HELD_OUT for t in f(x["note"])] vocabulary = set(train_tokens) out_of_vocab = [t for t in held_tokens if t not in vocabulary] extra = (f" number share {sum(map(is_number, out_of_vocab)) / len(out_of_vocab):.4f}" if number_share else "") print(f"{name:<28} {len(train_tokens) + len(held_tokens):>8} {len(vocabulary):>10}" f" {len(out_of_vocab):>17} {len(out_of_vocab) / len(held_tokens):>8.4f}{extra}") print(f"{'tokenization decision':<28} {'tokens':>8} {'vocabulary':>10}" f" {'out-of-vocabulary':>17} {'rate':>8}") measure(decision_1, "whitespace") measure(decision_2, "drop punctuation, lowercase") measure(decision_3, "+ stemming") def fold(f): return lambda note: ["" if is_number(t) else t for t in f(note)] print() measure(fold(decision_1), "whitespace", False) measure(fold(decision_2), "drop punctuation, lowercase", False) measure(fold(decision_3), "+ stemming", False) ``` ``` tokenization decision tokens vocabulary out-of-vocabulary rate whitespace 12044 748 225 0.0565 number share 0.9467 drop punctuation, lowercase 12044 669 225 0.0565 number share 0.9467 + stemming 12044 641 222 0.0557 number share 0.9595 whitespace 12044 203 12 0.0030 drop punctuation, lowercase 12044 124 12 0.0030 + stemming 12044 96 9 0.0023 ``` The three decisions produce the same 12,044 tokens but three different vocabularies: 748, 669, 641. The baseline vocabulary is inflated by 107 tokens, because `kapali` (closed) and `kapali,` and `Kapi` (door) and `kapi` are counted as distinct types. The out-of-vocabulary rate, on the other hand, is the same in the first two decisions: 0.0565. **Vocabulary size and out-of-vocabulary rate do not move in the same direction**, and the reason is written in the last column: 0.9467 of the out-of-vocabulary tokens are numbers. The reading value is different in every note, and a number that was not seen in training falls out of the vocabulary in the held-out set. When numbers are folded into a single token, the table reads differently: the vocabulary is 203, 124 and 96, the rate 0.0030 and 0.0023. That folding brings the vocabulary down to only 203 under the whitespace decision is a dependence between decisions: because the punctuation stays attached, `1107.` is not counted as a number. The triple can now be written: baseline 748 and 0.0565, all three decisions together 641 and 0.0557. The difference is 107 tokens in the vocabulary and 0.0008 in the out-of-vocabulary rate, and it **comes entirely from the representation**. In the number-folded table, stemming brings the out-of-vocabulary tokens down from 12 to 9; a difference of three tokens read over 3,985 tokens says nothing on its own. ## How Many Stems One Root Has Turkish is an agglutinative language: a root yields dozens of surface forms through successive suffixes. The aim of the stemmer is to collect them in a single stem; since the root mapping is at hand, how close it gets to the aim can be counted. ```python surface_forms = sorted({t for x in DATA for t in decision_2(x["note"]) if t in ROOT}) root_stems, stem_roots = {}, {} for t in surface_forms: root_stems.setdefault(ROOT[t], set()).add(stem(t)) stem_roots.setdefault(stem(t), set()).add(ROOT[t]) scattered = sorted(k for k, v in root_stems.items() if len(v) > 1) merging = sorted(g for g, v in stem_roots.items() if len(v) > 1) print(f"surface forms occurring in the notes {len(surface_forms)}, true roots {len(root_stems)}," f" stems produced by the stemmer {len(stem_roots)}") print(f"roots scattered over several stems {len(scattered)}, stems merging several roots" f" {len(merging)}") for k in scattered[:8]: print(f" {k:<10} -> {' '.join(sorted(root_stems[k]))}") for g in merging: print(f" {g:<10} <- {' '.join(sorted(stem_roots[g]))}") # The lemmatization table recognizes only the forms seen in the training set. TABLE = {t: ROOT[t] for x in TRAIN for t in decision_2(x["note"]) if t in ROOT} held_tokens = [t for x in HELD_OUT for t in decision_2(x["note"]) if not is_number(t)] print(f"\ntable recognizes {len(TABLE)} forms; coverage on non-number held-out tokens" f" {sum(t in TABLE for t in held_tokens) / len(held_tokens):.4f}") measure(fold(lambda note: [TABLE.get(t, t) for t in decision_2(note)]), "lemmatization (table)", False) measure(fold(decision_3), "stemming (rules)", False) ``` ``` surface forms occurring in the notes 134, true roots 85, stems produced by the stemmer 103 roots scattered over several stems 19, stems merging several roots 1 apartman -> apartma apartman baca -> bac baca bant -> ban bant bul -> bulun bulunma calis -> calisiyor calismiyor ev -> evd evin iz -> iz izi kapak -> kapag kapak kir <- kir kirli table recognizes 123 forms; coverage on non-number held-out tokens 0.9967 lemmatization (table) 12044 82 11 0.0028 stemming (rules) 12044 96 9 0.0023 ``` 134 surface forms occur in the notes, they belong to 85 roots, and the stemmer produces 103 stems: 18 stems away from the aim. The flaw runs in two directions. **Under-stemming** is seen in 19 of the 85 roots and comes from four causes: consonant softening (`kapak` (cover) and `kapagi` (its cover)), the longest match picking the wrong suffix (when `nin` is removed from `apartmanin` (of the apartment building), `apartma` is left), the last letter of the root being mistaken for a suffix (`baca` (chimney) and `bacada` (in the chimney)), and the condition of at least three characters (`izi` (its mark)). Verbs are never resolved, because the suffix list was written for noun inflection. **Over-stemming** happens once: removing `li` from `kirli` (dirty) merges it with `kir` (dirt), which is a separate root; `kapak kirli` (cover dirty) and `sayacta kir bulundu` (dirt was found in the meter) share the same stem. Lemmatization offers a different balance. The table recognizes the 123 forms seen in training and brings the vocabulary down to 82 tokens; this is smaller than the 96 of the rule and closer to the true number of roots. Its cost is in the last two rows: the coverage is 0.9967, a form that is not recognized passes through as it is, and the out-of-vocabulary tokens stay at 11, while the rule brings them down to 9. One is more accurate where it recognizes, the other works even where it does not. A detail lies under the measurement: the form `sayaci` (the meter; its meter) can be produced with both the accusative and the possessive suffix. No tokenization decision can tell them apart, because the decision looks only at characters. ## Summary - The notes are fictional and generated within the lesson: 1,800 notes, 85 roots, seed 20260218; the suspicion rate varies between 0.0486 and 0.7835 by kind, and the most frequent class holds a share of 0.7044. - Three tokenization decisions yield three different vocabularies from the same 12,044 tokens: 748, 669, 641. The domain baseline is splitting on whitespace, and it inflates the vocabulary by 107 tokens. - The out-of-vocabulary rate is the same in the first two decisions (0.0565), because 0.9467 of the out-of-vocabulary tokens are numbers; when numbers are folded, the vocabulary becomes 203, 124, 96 and the rate falls to 0.0030 and 0.0023. The difference comes entirely from the representation: there is no model, and the metric is the same on every row. - The stemmer splits 85 roots into 103 stems; 19 roots scatter over several stems, and one stem merges two roots. Table-based lemmatization brings the vocabulary down to 82 but has a coverage of 0.9967 and leaves the out-of-vocabulary tokens at 11. ## Next Step The text has been split into tokens, the forms have been reduced to stems and roots, and two metrics have been counted. What came out is still a list of words; not a single number that a model will see has been produced. The Data Preparation and Feature Engineering course had extracted five numbers by hand from the same note field and measured a contribution of 0.0317 on the held-out set; those five numbers are the baseline of this course. The next lesson turns the whole vocabulary into numbers: every token becomes a column, and term weighting is placed on top. The measure is how far above that line it goes.