Unit 01 of 02
Natural Language Processing
Counting what each decision adds when short Turkish field notes are turned into numbers: three tokenization decisions yielding three different vocabularies and out-of-vocabulary rates, and the agglutinative structure making stemming hard; what bag of words adds to hand-counted text features and what term weighting adds on top of it; embeddings being learned from context and analogy most often landing on the most frequent word; the same output giving a high number at the token level and a low one at the entity level, and the surprising height of the baseline that writes the most frequent label on every token; the share of each step in a classification pipeline; an embedding pretrained with masked prediction pulling ahead as labeled examples decrease; the high number that a procedure copying the input gets on an overlap-based metric, and the same output receiving four different rankings on four metrics.
Published Updated
Lessons
- 01 Text Preprocessing Measuring how much the decision moves the number when short Turkish field notes are split into tokens: the domain baseline of splitting on whitespace gives a vocabulary of 748 tokens and an out-of-vocabulary rate of 0.0565, while dropping punctuation and lowercasing brings the vocabulary to 669 and adding stemming brings it to 641 and the rate to 0.0557. 0.9467 of the out-of-vocabulary tokens are numbers; when numbers are folded into a single token the vocabulary becomes 203, 124 and 96. The rule-based stemmer splits 85 roots into 103 stems, scatters 19 roots over several stems and merges two roots in one stem; table-based lemmatization cuts the vocabulary to 82 but has a coverage of 0.9967 and can do nothing with a form it does not recognize. 9 min read
- 02 Numerical Representation Counting what each decision adds when short field notes are turned into columns, with the same model and the same held-out set: while the domain baseline of five hand-counted numbers gives 0.7383 and the most frequent class stays at 0.7017, a bag of words of 96 stems reaches 0.8217 and passes the line by 0.0833. Vocabulary size adds contribution from 8 to 64 and adds exactly zero from 64 to 96. Raw counts and inverse document frequency weighting do not move the accuracy at all (0.0833), dropping stop words loses 0.0067, and the ceiling that knows the event kind is 0.8250 — the bag is 0.0033 below the ceiling. 14 min read
- 03 Word Embeddings Giving every item of the vocabulary a vector learned from context and counting neighborhood against the setup's own phrase pools: while the expected rate of a random neighbor is 0.2543, the raw co-occurrence vector gives 0.4938 and the eight-dimensional learned embedding gives 0.6049. The total number misleads — 49 of the 81 forms are rare nouns that occur in a single pattern, and there even raw counts give 0.7755, while on frequent forms random is 0.0462, raw counts 0.0625 and the learned embedding 0.2500. Analogy holds in 0.0972 of 72 tests, and the ones that do not hold land not on the most frequent word but on the forms with the narrowest context. 15 min read
- 04 Sequence Labeling Taking the label from the document and giving it to every token separately, and measuring what accuracy counts: writing the most frequent label to every token, the domain baseline, gives a token accuracy of 0.5061 but an entity-level F1 of 0.0000 — not a single entity is found. The procedure that looks at the last three characters gives the same token accuracy of 0.9824 under both labeling schemes while it gives 0.9267 and 0.9600 at the entity level; the procedure that memorizes the token has a token accuracy of 1.0000 and stays at 0.9665 in the type scheme. In part of speech the baseline of the same form gives 0.5473 and does not mislead there. 15 min read
- 05 Text Classification Giving a single event kind decision to the whole of a field note and counting separately how many points each of the three steps of the pipeline adds: the most frequent class baseline is 0.4420, seven hand-counted measures give 0.5500, the bag of words with a linear model gives 0.9620. Measured over twelve setups, the representation step adds 0.3787 on average, the classifier step 0.0500, term weighting 0.0140; both tokenization decisions give a negative contribution, and suffix clipping costs 0.0165 on average while bringing the vocabulary down from 109 to 89. The contribution of a step is not a single number: weighting rises to 0.0340 with the class mean and falls to minus 0.0060 with the linear model. 15 min read
- 06 Pretraining and Fine-Tuning Carrying over to the labeled task an embedding table learned by masked prediction from unlabeled notes, and reading the difference it adds together with the number of labels: the proxy task raises its own baseline of 0.0419 to 0.3450 and spends zero labels for it. While the domain baseline is 0.4420, training from scratch with twenty labels gives 0.5850, pretraining and fine-tuning give 0.7480; training from scratch reaches this number only with forty labels. With one hundred sixty labels the difference turns to minus 0.0090. The control setup that does not learn — freezing a random table — stays below 0.51 in all five measurements and its two runs differ by 0.1320; the two runs of the pretrained table differ by 0.0020. 14 min read
- 07 Text Generation and Summarization Counting what the metric rewards on tasks where the output is text rather than a label: a procedure that gives the input back as it is and has learned nothing gets 1.0000 on the recall side of the overlap-based metric and ranks first among four procedures. The same procedure gets 0.6362 on the precision side and 0.7777 in balanced form, and falls to third place. When eight unrelated words are appended to the end of the copy, recall stays at 1.0000 while precision drops from 0.6362 to 0.2700. The generator procedure, which reports the content of the reference correctly but writes it with other words, gets 0.0684 in balanced form and 0.0114 on 2-grams; on the content check it is second with 0.9320. 14 min read
- 08 Natural Language Processing Metrics Reading a single output at four metrics and the four metrics giving four different rankings: the same four systems rank B > A > C > D at the token level, A > B > C > D at the entity level, C > B > A > D at the document level and D > B > A > C on the recall side of overlap, and the first of each metric is a different system. Each metric is read against its own baseline: the most frequent label at the token level 0.3970, the most frequent class at the document level 0.4420, marking no span at all at the entity level 0.0000, marking the whole input on the recall side of overlap 1.0000. In the last row the difference between the baseline and the best system is exactly 0.0000. 13 min read