--- title: 'Natural Language Processing and Computer Vision' source: 'https://academia.sh/en/courses/nlp-and-vision' language: en updated: '2026-10-21T05:03:26+00:00' license: 'CC BY-SA 4.0' lessons: 14 outcomes: - "Compares methods that turn text into a numerical representation against the field's baseline" - 'Separates image tasks into classification, detection and segmentation and builds the baseline of each' - 'Uses domain-specific evaluation metrics and shows with a number what the metric hides' --- # Natural Language Processing and Computer Vision Measuring what each step in turning text and images into numbers adds against the dumbest procedure in that field: the vocabulary size and out-of-vocabulary rate that the tokenization decision moves, what a bag of words and term weighting add to hand-counted features, what an embedding keeps and does not keep in neighborhood and analogy, token-level and entity-level metrics diverging, the share of each step in a classification pipeline, a proxy task pulling ahead as labeled examples decrease, the high number a procedure that copies its input gets on an overlap-based metric, four separate rankings of the same output on four metrics, the contribution of normalization and augmentation and the transform that corrupts the label, how many examples transfer adds with, the number a fixed box gets on the overlap metric and the threshold moving it, the number that assigning every pixel to the dominant class gets in segmentation, the inconsistency of frame decisions over time, and the matching success of a shared representation. ## Natural Language Processing Counting what each decision adds when short Turkish field notes are turned into numbers: three tokenization decisions yielding three different vocabularies and out-of-vocabulary rates, and the agglutinative structure making stemming hard; what bag of words adds to hand-counted text features and what term weighting adds on top of it; embeddings being learned from context and analogy most often landing on the most frequent word; the same output giving a high number at the token level and a low one at the entity level, and the surprising height of the baseline that writes the most frequent label on every token; the share of each step in a classification pipeline; an embedding pretrained with masked prediction pulling ahead as labeled examples decrease; the high number that a procedure copying the input gets on an overlap-based metric, and the same output receiving four different rankings on four metrics. ### Text Preprocessing Source: https://academia.sh/en/courses/nlp-and-vision/text-preprocessing Measuring how much the decision moves the number when short Turkish field notes are split into tokens: the domain baseline of splitting on whitespace gives a vocabulary of 748 tokens and an out-of-vocabulary rate of 0.0565, while dropping punctuation and lowercasing brings the vocabulary to 669 and adding stemming brings it to 641 and the rate to 0.0557. 0.9467 of the out-of-vocabulary tokens are numbers; when numbers are folded into a single token the vocabulary becomes 203, 124 and 96. The rule-based stemmer splits 85 roots into 103 stems, scatters 19 roots over several stems and merges two roots in one stem; table-based lemmatization cuts the vocabulary to 82 but has a coverage of 0.9967 and can do nothing with a form it does not recognize. The Deep Learning course built the architecture over twenty-four lessons, but the input was always a ready-made number: a column, a grid cell, a reading sequence, an event code. With a text in hand, the situation is different. In a note of a few words that the field crew files on a subscriber record, there is not a single number the model can read; what produces the number is a sequence of decisions. The measure of this course is set from here. Every lesson puts three numbers side by side: the **domain baseline**, the number of the most naive procedure on the same metric for that task; the **number of the method**; and the **difference**, written together with whether the difference came from the representation, the model or the metric. The rule is this: a domain metric cannot be read without the number of that domain's most naive procedure on the same metric. **A number whose domain baseline is not written is a number that has not been measured.** Here the baseline is splitting the text on whitespace, there is no model, and whatever the difference turns out to be comes from the representation alone. - **NL1.** The notes are **fictional**: they are generated within the lesson, seed 20260218, and none of them comes from a source system. The set is 1,800 notes and is the expanded version of the field-note column of the Data Preparation and Feature Engineering course. - **NL2.** The vocabulary is built from 85 roots: 24 inflected nouns, 10 verbs, 11 adjectives and 40 rare nouns. The weights of the rare nouns drop quickly; a few appear in the notes only once or twice. - **NL3.** The labels are **fictional**: the event kind (ordinary, fault, unreachable, suspected illicit use) and the binary illicit-use suspicion. The probability of suspicion depends only on the event kind. - **NL4.** A **token** is the smallest piece into which a text is split. In the JavaScript and TypeScript course, the unit produced by the language parser was called a **lexical token**; the distinction is kept, because there the split is exact by grammar and here it is a decision. - **NL5.** The first 1,200 notes are the training set and the remaining 600 notes are the held-out set. The vocabulary is built **from the training set only**; a token that is seen in the held-out set and is not in the vocabulary is **out-of-vocabulary**. - **NL6.** Text normalization and string procedures were measured in the data cleaning part of the Data Collection and Preparation course; the procedure is not repeated, only the number that the decision moves is measured. - **NL7.** The metric is vocabulary size and out-of-vocabulary rate (smaller is better) and it is the same on every row. - **NL8.** The accuracy of the stemmer is measured against the root mapping of the setup; the lemmatization table is **only the part of the same mapping that was seen in the training set**. #### The Setup of the Notes A note consists of one or a few short phrases: the first phrase carries the event kind, and the extra phrase and the rare-noun phrase are independent of the kind and are noise. ```python # COMMON — FICTIONAL. The field notes are generated within the lesson, seed 20260218; # they do not come from a source system. The vocabulary has 45 inflected roots and # 40 rare nouns. SEED, NOTE_COUNT, M32 = 20260218, 1800, 0xFFFFFFFF NOUNS = { # 0 base, 1 genitive, 2 locative, 3 possessive, 4 plural "sayac": "sayac sayacin sayacta sayaci sayaclar", "kapak": "kapak kapagin kapakta kapagi kapaklar", "muhur": "muhur muhurun muhurde muhuru muhurler", "vana": "vana vananin vanada vanasi vanalar", "kelepce": "kelepce kelepcenin kelepcede kelepcesi kelepceler", "boru": "boru borunun boruda borusu borular", "pano": "pano panonun panoda panosu panolar", "kapi": "kapi kapinin kapida kapisi kapilar", "zil": "zil zilin zilde zili ziller", "bahce": "bahce bahcenin bahcede bahcesi bahceler", "apartman": "apartman apartmanin apartmanda apartmani apartmanlar", "bodrum": "bodrum bodrumun bodrumda bodrumu bodrumlar", "giris": "giris girisin giriste girisi girisler", "adres": "adres adresin adreste adresi adresler", "okuma": "okuma okumanin okumada okumasi okumalar", "olcum": "olcum olcumun olcumde olcumu olcumler", "erisim": "erisim erisimin erisimde erisimi erisimler", "ziyaret": "ziyaret ziyaretin ziyarette ziyareti ziyaretler", "abone": "abone abonenin abonede abonesi aboneler", "arac": "arac aracin aracta araci araclar", "kir": "kir kirin kirde kiri kirler", "iz": "iz izin izde izi izler", "sokak": "sokak sokagin sokakta sokagi sokaklar", "ev": "ev evin evde evi evler", } ADJECTIVES = "kirik gevsek kirli normal bozuk acik kapali eksik temiz yeni yok".split() VERBS = { # 0 passive past, 1 negative "calis": "calisiyor calismiyor", "don": "donuyor donmuyor", "oku": "okundu okunmadi", "ac": "acildi acilmadi", "gir": "girildi girilemedi", "ulas": "ulasildi ulasilamadi", "tekrarla": "tekrarlandi tekrarlanmadi", "bul": "bulundu bulunmadi", "sagla": "saglandi saglanamadi", "degil": "degil degildi", } RARE_NOUNS = """kablo kablosu kabloda|cam cami camda|tel teli telde|mandal mandali mandalda merdiven merdiveni merdivende|tahta tahtasi tahtada|cukur cukuru cukurda|kar kari karda buz buzu buzda|cop copu copta|duvar duvari duvarda|kutu kutusu kutuda|vida vidasi vidada conta contasi contada|filtre filtresi filtrede|bant banti bantta|etiket etiketi etikette kilit kiliti kilitte|asansor asansoru asansorde|zemin zemini zeminde|tavan tavani tavanda kuyu kuyusu kuyuda|depo deposu depoda|ocak ocagi ocakta|direk diregi direkte kanal kanali kanalda|numara numarasi numarada|cati catisi catida|baca bacasi bacada kova kovasi kovada|levha levhasi levhada|somun somunu somunda|tapa tapasi tapada izolasyon izolasyonu izolasyonda|supap supabi supapta|menfez menfezi menfezde kayit kaydi kayitta|temel temeli temelde|kose kosesi kosede|koruma korumasi korumada""" PHRASES = { "ordinary": ["{sayac.0} {calis.0}", "{okuma.0} normal", "{olcum.0} {tekrarla.0}", "{sayac.1} {erisim.3} {sagla.0}", "{kapak.0} temiz", "{vana.0} acik", "{sayac.2} {iz.0} {bul.1}", "{olcum.0} icin {erisim.0} {sagla.0}", "{sayac.1} {kapak.3} yeni", "{muhur.0} ile {kapak.0} temiz"], "fault": ["{sayac.0} {don.1}", "{kapak.0} kirik", "{pano.0} bozuk", "{sayac.1} {kapak.3} kirik", "{olcum.0} {oku.1}", "{vana.0} bozuk", "{sayac.2} {kir.0} {bul.0}", "{sayac.4} {calis.1}", "{sayac.0} {calis.0} ama {okuma.0} yok", "{pano.1} {kapak.3} eksik"], "unreachable": ["{kapi.0} {ac.1}", "{zil.0} {calis.1}", "{abone.0} {ev.2} {degil.1}", "{erisim.0} {sagla.1}", "{bodrum.0} {kapi.3} kapali", "{apartman.1} {giris.3} kapali", "{adres.2} {ulas.1}", "{bahce.1} {kapi.3} kapali", "{apartman.2} {zil.0} {calis.1}", "{sayac.2} {erisim.0} yok"], "illicit": ["{muhur.0} kirik", "{kelepce.0} gevsek", "{muhur.2} {iz.0} {bul.0}", "{sayac.1} {muhur.3} kirik", "{kelepce.2} {iz.0} {bul.0}", "{muhur.0} eksik", "{vana.1} {kelepce.3} gevsek", "{muhur.1} {iz.3} {bul.0}", "{boru.2} {kelepce.0} eksik", "{kapak.2} bir {iz.0} {bul.0}"], } EXTRA_PHRASES = ["{ziyaret.0} {tekrarla.0}", "{sokak.2} {arac.0} {bul.0}", "{adres.0} yeni", "{bahce.1} {kapi.3} acik", "{apartman.2} {gir.0}", "{ev.1} {kapi.3} acik", "{kapak.0} kirli", "{olcum.0} {tekrarla.1}", "{arac.0} {sokak.2}", "bu {ziyaret.2} {olcum.0} {tekrarla.0}"] KINDS = [("ordinary", 0.36, 0.04), ("fault", 0.26, 0.28), ("unreachable", 0.16, 0.15), ("illicit", 0.22, 0.82)] FORMS = {k: v.split() for d in (NOUNS, VERBS) for k, v in d.items()} RARE = [s.split() for s in RARE_NOUNS.replace("\n", "|").split("|")] WEIGHTS = [1 / (i + 1) ** 1.6 for i in range(len(RARE))] WEIGHTS = [w / sum(WEIGHTS) for w in WEIGHTS] ROOT = {f: k for k, fs in FORMS.items() for f in fs} ROOT.update({f: fs[0] for fs in RARE for f in fs}) ROOT.update({w: w for w in ADJECTIVES}) STOP_WORDS = ["ve", "ile", "bir", "bu", "ama", "icin"] ROOT.update({w: w for w in STOP_WORDS}) def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def render_phrase(template): parts = [] for part in template.split(): if part.startswith("{"): name, _, i = part[1:-1].partition(".") parts.append(FORMS[name][int(i)]) else: parts.append(part) return " ".join(parts) DATA = [] for i in range(NOTE_COUNT): r = rng(SEED + i) kind = KINDS[pick(r(), [x[1] for x in KINDS])] clauses = [render_phrase(PHRASES[kind[0]][int(r() * 10)])] if r() < 0.55: clauses.append(render_phrase(EXTRA_PHRASES[int(r() * 10)])) if r() < 0.30: clauses.append(render_phrase(PHRASES[kind[0]][int(r() * 10)])) if r() < 0.22: forms, adjective = RARE[pick(r(), WEIGHTS)], ADJECTIVES[int(r() * len(ADJECTIVES))] clauses.append(forms[2] + " kir bulundu" if r() < 0.34 else forms[int(r() * 2)] + " " + adjective) note = clauses[0] for x in clauses[1:]: note += (" ve " if r() < 0.22 else ", ") + x if r() < 0.52: note += ", okuma " + str(240 + int(r() * 1760)) DATA.append({"id": 10001 + i, "note": note[0].upper() + note[1:] + ".", "kind": kind[0], "suspicious": int(r() < kind[2])}) TRAIN, HELD_OUT = DATA[:1200], DATA[1200:] for i in (0, 3, 41): print(f"{DATA[i]['kind']:<12} {DATA[i]['suspicious']} {DATA[i]['note']}") print(f"\n{len(DATA)} notes, {len(ROOT)} vocabulary forms, " f"{len(FORMS) + len(RARE) + len(ADJECTIVES)} roots") for name, _, _ in KINDS: n = [x for x in DATA if x["kind"] == name] print(f"{name:<12} {len(n):>5} notes suspicion rate " f"{sum(x['suspicious'] for x in n) / len(n):.4f}") print(f"{'total':<12} {len(DATA):>5} notes suspicion rate " f"{sum(x['suspicious'] for x in DATA) / len(DATA):.4f}") ``` ``` illicit 1 Vananin kelepcesi gevsek. ordinary 0 Olcum icin erisim saglandi, sayacin kapagi yeni, mandalda kir bulundu. illicit 1 Boruda kelepce eksik, okuma 1505. 1800 notes, 277 vocabulary forms, 85 roots ordinary 638 notes suspicion rate 0.0486 fault 446 notes suspicion rate 0.3072 unreachable 305 notes suspicion rate 0.1377 illicit 411 notes suspicion rate 0.7835 total 1800 notes suspicion rate 0.2956 ``` The suspicion rate varies between 0.0486 and 0.7835 by kind: the text is related to the label but does not determine it, and 89 of the 411 notes that carry an illicit phrase have no suspicion. This is the ceiling for the later lessons, and the most frequent class holds a share of 0.7044. #### Three Tokenization Decisions The most naive procedure is to split on whitespace: punctuation stays attached to the word, and a capital letter at the start of a sentence counts as a separate type. The second decision drops the punctuation and lowercases. The third decision adds rule-based **stemming** on top: the longest suffix matching at the end is removed, provided that at least three characters remain. ```python SUFFIXES = ["lari", "leri", "lar", "ler", "nin", "nun", "dan", "den", "tan", "ten", "dir", "tir", "dur", "da", "de", "ta", "te", "in", "un", "yi", "yu", "si", "su", "li", "lu", "di", "du", "ti", "tu", "i", "u", "a", "e"] def decision_1(note): # whitespace return note.split() def decision_2(note): # punctuation dropped, lowercased return [t for t in (s.strip(",.").lower() for s in note.split()) if t] def stem(token): for suffix in SUFFIXES: if token.endswith(suffix) and len(token) - len(suffix) >= 3: return token[:-len(suffix)] return token def decision_3(note): # + stemming return [stem(t) for t in decision_2(note)] def is_number(token): return token.strip(",.").isdigit() def measure(f, name, number_share=True): train_tokens = [t for x in TRAIN for t in f(x["note"])] held_tokens = [t for x in HELD_OUT for t in f(x["note"])] vocabulary = set(train_tokens) out_of_vocab = [t for t in held_tokens if t not in vocabulary] extra = (f" number share {sum(map(is_number, out_of_vocab)) / len(out_of_vocab):.4f}" if number_share else "") print(f"{name:<28} {len(train_tokens) + len(held_tokens):>8} {len(vocabulary):>10}" f" {len(out_of_vocab):>17} {len(out_of_vocab) / len(held_tokens):>8.4f}{extra}") print(f"{'tokenization decision':<28} {'tokens':>8} {'vocabulary':>10}" f" {'out-of-vocabulary':>17} {'rate':>8}") measure(decision_1, "whitespace") measure(decision_2, "drop punctuation, lowercase") measure(decision_3, "+ stemming") def fold(f): return lambda note: ["" if is_number(t) else t for t in f(note)] print() measure(fold(decision_1), "whitespace", False) measure(fold(decision_2), "drop punctuation, lowercase", False) measure(fold(decision_3), "+ stemming", False) ``` ``` tokenization decision tokens vocabulary out-of-vocabulary rate whitespace 12044 748 225 0.0565 number share 0.9467 drop punctuation, lowercase 12044 669 225 0.0565 number share 0.9467 + stemming 12044 641 222 0.0557 number share 0.9595 whitespace 12044 203 12 0.0030 drop punctuation, lowercase 12044 124 12 0.0030 + stemming 12044 96 9 0.0023 ``` The three decisions produce the same 12,044 tokens but three different vocabularies: 748, 669, 641. The baseline vocabulary is inflated by 107 tokens, because `kapali` (closed) and `kapali,` and `Kapi` (door) and `kapi` are counted as distinct types. The out-of-vocabulary rate, on the other hand, is the same in the first two decisions: 0.0565. **Vocabulary size and out-of-vocabulary rate do not move in the same direction**, and the reason is written in the last column: 0.9467 of the out-of-vocabulary tokens are numbers. The reading value is different in every note, and a number that was not seen in training falls out of the vocabulary in the held-out set. When numbers are folded into a single token, the table reads differently: the vocabulary is 203, 124 and 96, the rate 0.0030 and 0.0023. That folding brings the vocabulary down to only 203 under the whitespace decision is a dependence between decisions: because the punctuation stays attached, `1107.` is not counted as a number. The triple can now be written: baseline 748 and 0.0565, all three decisions together 641 and 0.0557. The difference is 107 tokens in the vocabulary and 0.0008 in the out-of-vocabulary rate, and it **comes entirely from the representation**. In the number-folded table, stemming brings the out-of-vocabulary tokens down from 12 to 9; a difference of three tokens read over 3,985 tokens says nothing on its own. #### How Many Stems One Root Has Turkish is an agglutinative language: a root yields dozens of surface forms through successive suffixes. The aim of the stemmer is to collect them in a single stem; since the root mapping is at hand, how close it gets to the aim can be counted. ```python surface_forms = sorted({t for x in DATA for t in decision_2(x["note"]) if t in ROOT}) root_stems, stem_roots = {}, {} for t in surface_forms: root_stems.setdefault(ROOT[t], set()).add(stem(t)) stem_roots.setdefault(stem(t), set()).add(ROOT[t]) scattered = sorted(k for k, v in root_stems.items() if len(v) > 1) merging = sorted(g for g, v in stem_roots.items() if len(v) > 1) print(f"surface forms occurring in the notes {len(surface_forms)}, true roots {len(root_stems)}," f" stems produced by the stemmer {len(stem_roots)}") print(f"roots scattered over several stems {len(scattered)}, stems merging several roots" f" {len(merging)}") for k in scattered[:8]: print(f" {k:<10} -> {' '.join(sorted(root_stems[k]))}") for g in merging: print(f" {g:<10} <- {' '.join(sorted(stem_roots[g]))}") # The lemmatization table recognizes only the forms seen in the training set. TABLE = {t: ROOT[t] for x in TRAIN for t in decision_2(x["note"]) if t in ROOT} held_tokens = [t for x in HELD_OUT for t in decision_2(x["note"]) if not is_number(t)] print(f"\ntable recognizes {len(TABLE)} forms; coverage on non-number held-out tokens" f" {sum(t in TABLE for t in held_tokens) / len(held_tokens):.4f}") measure(fold(lambda note: [TABLE.get(t, t) for t in decision_2(note)]), "lemmatization (table)", False) measure(fold(decision_3), "stemming (rules)", False) ``` ``` surface forms occurring in the notes 134, true roots 85, stems produced by the stemmer 103 roots scattered over several stems 19, stems merging several roots 1 apartman -> apartma apartman baca -> bac baca bant -> ban bant bul -> bulun bulunma calis -> calisiyor calismiyor ev -> evd evin iz -> iz izi kapak -> kapag kapak kir <- kir kirli table recognizes 123 forms; coverage on non-number held-out tokens 0.9967 lemmatization (table) 12044 82 11 0.0028 stemming (rules) 12044 96 9 0.0023 ``` 134 surface forms occur in the notes, they belong to 85 roots, and the stemmer produces 103 stems: 18 stems away from the aim. The flaw runs in two directions. **Under-stemming** is seen in 19 of the 85 roots and comes from four causes: consonant softening (`kapak` (cover) and `kapagi` (its cover)), the longest match picking the wrong suffix (when `nin` is removed from `apartmanin` (of the apartment building), `apartma` is left), the last letter of the root being mistaken for a suffix (`baca` (chimney) and `bacada` (in the chimney)), and the condition of at least three characters (`izi` (its mark)). Verbs are never resolved, because the suffix list was written for noun inflection. **Over-stemming** happens once: removing `li` from `kirli` (dirty) merges it with `kir` (dirt), which is a separate root; `kapak kirli` (cover dirty) and `sayacta kir bulundu` (dirt was found in the meter) share the same stem. Lemmatization offers a different balance. The table recognizes the 123 forms seen in training and brings the vocabulary down to 82 tokens; this is smaller than the 96 of the rule and closer to the true number of roots. Its cost is in the last two rows: the coverage is 0.9967, a form that is not recognized passes through as it is, and the out-of-vocabulary tokens stay at 11, while the rule brings them down to 9. One is more accurate where it recognizes, the other works even where it does not. A detail lies under the measurement: the form `sayaci` (the meter; its meter) can be produced with both the accusative and the possessive suffix. No tokenization decision can tell them apart, because the decision looks only at characters. #### Summary - The notes are fictional and generated within the lesson: 1,800 notes, 85 roots, seed 20260218; the suspicion rate varies between 0.0486 and 0.7835 by kind, and the most frequent class holds a share of 0.7044. - Three tokenization decisions yield three different vocabularies from the same 12,044 tokens: 748, 669, 641. The domain baseline is splitting on whitespace, and it inflates the vocabulary by 107 tokens. - The out-of-vocabulary rate is the same in the first two decisions (0.0565), because 0.9467 of the out-of-vocabulary tokens are numbers; when numbers are folded, the vocabulary becomes 203, 124, 96 and the rate falls to 0.0030 and 0.0023. The difference comes entirely from the representation: there is no model, and the metric is the same on every row. - The stemmer splits 85 roots into 103 stems; 19 roots scatter over several stems, and one stem merges two roots. Table-based lemmatization brings the vocabulary down to 82 but has a coverage of 0.9967 and leaves the out-of-vocabulary tokens at 11. #### Next Step The text has been split into tokens, the forms have been reduced to stems and roots, and two metrics have been counted. What came out is still a list of words; not a single number that a model will see has been produced. The Data Preparation and Feature Engineering course had extracted five numbers by hand from the same note field and measured a contribution of 0.0317 on the held-out set; those five numbers are the baseline of this course. The next lesson turns the whole vocabulary into numbers: every token becomes a column, and term weighting is placed on top. The measure is how far above that line it goes. ### Numerical Representation Source: https://academia.sh/en/courses/nlp-and-vision/numerical-representation Counting what each decision adds when short field notes are turned into columns, with the same model and the same held-out set: while the domain baseline of five hand-counted numbers gives 0.7383 and the most frequent class stays at 0.7017, a bag of words of 96 stems reaches 0.8217 and passes the line by 0.0833. Vocabulary size adds contribution from 8 to 64 and adds exactly zero from 64 to 96. Raw counts and inverse document frequency weighting do not move the accuracy at all (0.0833), dropping stop words loses 0.0067, and the ceiling that knows the event kind is 0.8250 — the bag is 0.0033 below the ceiling. The previous lesson split the text into tokens and brought the vocabulary down to 96 stems. What came out was still a list of words; not a single number that a model would see had been produced. This lesson turns the vocabulary into columns: every stem becomes a column, and whether it occurs in the note becomes the value of that column. The name of the procedure is **bag of words**, because the order is discarded — `muhur kirik` (seal broken) and `kirik muhur` (broken seal) give the same row. The line of the measure is ready. The Data Preparation and Feature Engineering course had extracted five numbers by hand from the same note field, and those five numbers had added 0.0317 on the held-out set to a decision tree. The **domain baseline** is those five numbers: the cheapest way of extracting numbers from a text. How far above this line does the bag of words go, what does the contribution do as the vocabulary grows, and what does term weighting put on top. - **NL9.** The setup, the set and the seed are as in `nlp/01`. The tokenization decision is the third decision of 01 — dropping punctuation, lowercasing, stemming, number folding — and it is **fixed** in this lesson. - **NL10.** The task is the binary prediction of illicit-use suspicion. The metric is the accuracy on the held-out set (larger is better), and the recall of the suspicious class is written next to it. The metric definitions were set up in the Supervised Learning course and are not repeated. - **NL11.** All representations use **the same model**: a single-layer linear model, full-batch gradient descent, 120 rounds, step 1.5, threshold zero. Because the model and the metric are fixed, the difference can come only from the representation. - **NL12.** Every step of the pipeline is learned **from the training set only**: the vocabulary, the document frequency, the mean and the standard deviation. The rule of the Data Preparation and Feature Engineering course continues. - **NL13.** The domain baseline is **five hand-counted numbers**: note length, word count and three keyword indicators (seal, access, vehicle). The first two are scaled with the mean and standard deviation of the training set. The most frequent class is the absolute baseline, and its recall is zero. - **NL14.** The vocabulary size sweep is six candidates: 8, 16, 32, 48, 64, 96. The vocabulary is taken in order from the most frequent stems of the training set. - **NL15.** Term weighting is named by its **form**: raw count, and the count multiplied by the inverse document frequency — the second is brought to unit length per document. - **NL16.** The ceiling is the number of a procedure that knows the event kind on the same metric; since the label is generated from the kind alone, this is the highest number that can be read from the text. #### Setup The notes are regenerated from the same seed as the setup of `nlp/01`. ```python # COMMON — FICTIONAL. The field notes are generated within the lesson, seed 20260218; # they do not come from a source system. The vocabulary has 45 inflected roots and # 40 rare nouns. SEED, NOTE_COUNT, M32 = 20260218, 1800, 0xFFFFFFFF NOUNS = { # 0 base, 1 genitive, 2 locative, 3 possessive, 4 plural "sayac": "sayac sayacin sayacta sayaci sayaclar", "kapak": "kapak kapagin kapakta kapagi kapaklar", "muhur": "muhur muhurun muhurde muhuru muhurler", "vana": "vana vananin vanada vanasi vanalar", "kelepce": "kelepce kelepcenin kelepcede kelepcesi kelepceler", "boru": "boru borunun boruda borusu borular", "pano": "pano panonun panoda panosu panolar", "kapi": "kapi kapinin kapida kapisi kapilar", "zil": "zil zilin zilde zili ziller", "bahce": "bahce bahcenin bahcede bahcesi bahceler", "apartman": "apartman apartmanin apartmanda apartmani apartmanlar", "bodrum": "bodrum bodrumun bodrumda bodrumu bodrumlar", "giris": "giris girisin giriste girisi girisler", "adres": "adres adresin adreste adresi adresler", "okuma": "okuma okumanin okumada okumasi okumalar", "olcum": "olcum olcumun olcumde olcumu olcumler", "erisim": "erisim erisimin erisimde erisimi erisimler", "ziyaret": "ziyaret ziyaretin ziyarette ziyareti ziyaretler", "abone": "abone abonenin abonede abonesi aboneler", "arac": "arac aracin aracta araci araclar", "kir": "kir kirin kirde kiri kirler", "iz": "iz izin izde izi izler", "sokak": "sokak sokagin sokakta sokagi sokaklar", "ev": "ev evin evde evi evler", } ADJECTIVES = "kirik gevsek kirli normal bozuk acik kapali eksik temiz yeni yok".split() VERBS = { # 0 passive past, 1 negative "calis": "calisiyor calismiyor", "don": "donuyor donmuyor", "oku": "okundu okunmadi", "ac": "acildi acilmadi", "gir": "girildi girilemedi", "ulas": "ulasildi ulasilamadi", "tekrarla": "tekrarlandi tekrarlanmadi", "bul": "bulundu bulunmadi", "sagla": "saglandi saglanamadi", "degil": "degil degildi", } RARE_NOUNS = """kablo kablosu kabloda|cam cami camda|tel teli telde|mandal mandali mandalda merdiven merdiveni merdivende|tahta tahtasi tahtada|cukur cukuru cukurda|kar kari karda buz buzu buzda|cop copu copta|duvar duvari duvarda|kutu kutusu kutuda|vida vidasi vidada conta contasi contada|filtre filtresi filtrede|bant banti bantta|etiket etiketi etikette kilit kiliti kilitte|asansor asansoru asansorde|zemin zemini zeminde|tavan tavani tavanda kuyu kuyusu kuyuda|depo deposu depoda|ocak ocagi ocakta|direk diregi direkte kanal kanali kanalda|numara numarasi numarada|cati catisi catida|baca bacasi bacada kova kovasi kovada|levha levhasi levhada|somun somunu somunda|tapa tapasi tapada izolasyon izolasyonu izolasyonda|supap supabi supapta|menfez menfezi menfezde kayit kaydi kayitta|temel temeli temelde|kose kosesi kosede|koruma korumasi korumada""" PHRASES = { "ordinary": ["{sayac.0} {calis.0}", "{okuma.0} normal", "{olcum.0} {tekrarla.0}", "{sayac.1} {erisim.3} {sagla.0}", "{kapak.0} temiz", "{vana.0} acik", "{sayac.2} {iz.0} {bul.1}", "{olcum.0} icin {erisim.0} {sagla.0}", "{sayac.1} {kapak.3} yeni", "{muhur.0} ile {kapak.0} temiz"], "fault": ["{sayac.0} {don.1}", "{kapak.0} kirik", "{pano.0} bozuk", "{sayac.1} {kapak.3} kirik", "{olcum.0} {oku.1}", "{vana.0} bozuk", "{sayac.2} {kir.0} {bul.0}", "{sayac.4} {calis.1}", "{sayac.0} {calis.0} ama {okuma.0} yok", "{pano.1} {kapak.3} eksik"], "unreachable": ["{kapi.0} {ac.1}", "{zil.0} {calis.1}", "{abone.0} {ev.2} {degil.1}", "{erisim.0} {sagla.1}", "{bodrum.0} {kapi.3} kapali", "{apartman.1} {giris.3} kapali", "{adres.2} {ulas.1}", "{bahce.1} {kapi.3} kapali", "{apartman.2} {zil.0} {calis.1}", "{sayac.2} {erisim.0} yok"], "illicit": ["{muhur.0} kirik", "{kelepce.0} gevsek", "{muhur.2} {iz.0} {bul.0}", "{sayac.1} {muhur.3} kirik", "{kelepce.2} {iz.0} {bul.0}", "{muhur.0} eksik", "{vana.1} {kelepce.3} gevsek", "{muhur.1} {iz.3} {bul.0}", "{boru.2} {kelepce.0} eksik", "{kapak.2} bir {iz.0} {bul.0}"], } EXTRA_PHRASES = ["{ziyaret.0} {tekrarla.0}", "{sokak.2} {arac.0} {bul.0}", "{adres.0} yeni", "{bahce.1} {kapi.3} acik", "{apartman.2} {gir.0}", "{ev.1} {kapi.3} acik", "{kapak.0} kirli", "{olcum.0} {tekrarla.1}", "{arac.0} {sokak.2}", "bu {ziyaret.2} {olcum.0} {tekrarla.0}"] KINDS = [("ordinary", 0.36, 0.04), ("fault", 0.26, 0.28), ("unreachable", 0.16, 0.15), ("illicit", 0.22, 0.82)] FORMS = {k: v.split() for d in (NOUNS, VERBS) for k, v in d.items()} RARE = [s.split() for s in RARE_NOUNS.replace("\n", "|").split("|")] WEIGHTS = [1 / (i + 1) ** 1.6 for i in range(len(RARE))] WEIGHTS = [w / sum(WEIGHTS) for w in WEIGHTS] ROOT = {f: k for k, fs in FORMS.items() for f in fs} ROOT.update({f: fs[0] for fs in RARE for f in fs}) ROOT.update({w: w for w in ADJECTIVES}) STOP_WORDS = ["ve", "ile", "bir", "bu", "ama", "icin"] ROOT.update({w: w for w in STOP_WORDS}) def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def render_phrase(template): parts = [] for part in template.split(): if part.startswith("{"): name, _, i = part[1:-1].partition(".") parts.append(FORMS[name][int(i)]) else: parts.append(part) return " ".join(parts) DATA = [] for i in range(NOTE_COUNT): r = rng(SEED + i) kind = KINDS[pick(r(), [x[1] for x in KINDS])] clauses = [render_phrase(PHRASES[kind[0]][int(r() * 10)])] if r() < 0.55: clauses.append(render_phrase(EXTRA_PHRASES[int(r() * 10)])) if r() < 0.30: clauses.append(render_phrase(PHRASES[kind[0]][int(r() * 10)])) if r() < 0.22: forms, adjective = RARE[pick(r(), WEIGHTS)], ADJECTIVES[int(r() * len(ADJECTIVES))] clauses.append(forms[2] + " kir bulundu" if r() < 0.34 else forms[int(r() * 2)] + " " + adjective) note = clauses[0] for x in clauses[1:]: note += (" ve " if r() < 0.22 else ", ") + x if r() < 0.52: note += ", okuma " + str(240 + int(r() * 1760)) DATA.append({"id": 10001 + i, "note": note[0].upper() + note[1:] + ".", "kind": kind[0], "suspicious": int(r() < kind[2])}) TRAIN, HELD_OUT = DATA[:1200], DATA[1200:] print(f"{len(DATA)} fictional notes, {len(FORMS) + len(RARE) + len(ADJECTIVES)} roots, " f"suspicion rate {sum(x['suspicious'] for x in DATA) / len(DATA):.4f}") print(DATA[3]["note"]) ``` ``` 1800 fictional notes, 85 roots, suspicion rate 0.2956 Olcum icin erisim saglandi, sayacin kapagi yeni, mandalda kir bulundu. ``` #### Bag of Words and Vocabulary Size The model is written once and stays the same on every row. The features are given sparsely: because a note contains a dozen or so stems, a few multiplications per row are enough even when the column count goes up to 96. ```python import math from collections import Counter SUFFIXES = ["lari", "leri", "lar", "ler", "nin", "nun", "dan", "den", "tan", "ten", "dir", "tir", "dur", "da", "de", "ta", "te", "in", "un", "yi", "yu", "si", "su", "li", "lu", "di", "du", "ti", "tu", "i", "u", "a", "e"] def stem(token): for suffix in SUFFIXES: if token.endswith(suffix) and len(token) - len(suffix) >= 3: return token[:-len(suffix)] return token for x in DATA: # third decision of nlp/01: drop punctuation, lowercase, stem, fold numbers x["tokens"] = [("" if t.isdigit() else stem(t)) for t in (s.strip(",.").lower() for s in x["note"].split()) if t] def fit_model(features, y, size, rounds=120, step=1.5): w = [0.0] * (size + 1) for _ in range(rounds): grad = [0.0] * (size + 1) for x, target in zip(features, y): z = w[size] + sum(w[i] * v for i, v in x) err = 1 / (1 + math.exp(-max(-30.0, min(30.0, z)))) - target for i, v in x: grad[i] += err * v grad[size] += err for i in range(size + 1): w[i] -= step * grad[i] / len(y) return w def evaluate(w, features, y, size): correct = caught = 0 for x, truth in zip(features, y): pred = int(w[size] + sum(w[i] * v for i, v in x) > 0) correct, caught = correct + (pred == truth), caught + (pred == 1 and truth == 1) return correct / len(y), caught / sum(y) def run(name, build, size): # the first call sets the line y_train, y_held = [x["suspicious"] for x in TRAIN], [x["suspicious"] for x in HELD_OUT] w = fit_model([build(x) for x in TRAIN], y_train, size) acc_train, _ = evaluate(w, [build(x) for x in TRAIN], y_train, size) acc_held, recall_held = evaluate(w, [build(x) for x in HELD_OUT], y_held, size) if not BASELINE: BASELINE.append(acc_held) print(f"{name:<26} {size:>7} {acc_train:>9.4f} {acc_held:>9.4f} {recall_held:>10.4f}" f" {acc_held - BASELINE[0]:>+8.4f}") return acc_held KEYWORDS, BASELINE = ["muhur", "erisim", "arac"], [] raw = [[len(x["note"]), len(x["tokens"])] for x in TRAIN] MEAN = [sum(r[i] for r in raw) / len(raw) for i in (0, 1)] STD = [math.sqrt(sum((r[i] - MEAN[i]) ** 2 for r in raw) / len(raw)) for i in (0, 1)] def by_hand(x): # the five numbers of the Data Preparation and Feature Engineering course o = [(0, (len(x["note"]) - MEAN[0]) / STD[0]), (1, (len(x["tokens"]) - MEAN[1]) / STD[1])] for i, keyword in enumerate(KEYWORDS): o.append((2 + i, float(any(t.startswith(keyword) for t in x["tokens"])))) return o MAJORITY = sum(x["suspicious"] == 0 for x in HELD_OUT) / len(HELD_OUT) print(f"{'representation':<26} {'columns':>7} {'train':>9} {'held-out':>9} {'recall':>10}" f" {'vs line':>8}") run("hand-counted five numbers", by_hand, 5) print(f"{'most frequent class':<26} {0:>7} {'':>9} {MAJORITY:>9.4f} {0.0:>10.4f}" f" {MAJORITY - BASELINE[0]:>+8.4f}") FREQ = Counter(t for x in TRAIN for t in x["tokens"]) RANKED = sorted(FREQ, key=lambda t: (-FREQ[t], t)) DOC_FREQ = Counter(t for x in TRAIN for t in set(x["tokens"])) STOP_STEMS = {stem(w) for w in STOP_WORDS} def bag(terms, mode="indicator"): index = {t: i for i, t in enumerate(terms)} idf = {t: math.log(len(TRAIN) / DOC_FREQ.get(t, 1)) for t in index} def build(x): counts = Counter(t for t in x["tokens"] if t in index) if mode == "indicator": return [(index[t], 1.0) for t in counts] if mode == "count": return [(index[t], float(v)) for t, v in counts.items()] norm = math.sqrt(sum((v * idf[t]) ** 2 for t, v in counts.items())) or 1.0 return [(index[t], v * idf[t] / norm) for t, v in counts.items()] return build print() for n in (8, 16, 32, 48, 64, 96): run(f"bag of words {n}", bag(RANKED[:n]), n) ``` ``` representation columns train held-out recall vs line hand-counted five numbers 5 0.7608 0.7383 0.3408 +0.0000 most frequent class 0 0.7017 0.0000 -0.0367 bag of words 8 8 0.7783 0.7667 0.3296 +0.0283 bag of words 16 16 0.7908 0.7700 0.3911 +0.0317 bag of words 32 32 0.8208 0.8067 0.5140 +0.0683 bag of words 48 48 0.8300 0.8133 0.5363 +0.0750 bag of words 64 64 0.8333 0.8217 0.5698 +0.0833 bag of words 96 96 0.8350 0.8217 0.5698 +0.0833 ``` The line is 0.7383, and it passes the most frequent class by 0.0367. In the Data Preparation and Feature Engineering course, the same five numbers had added 0.0317 to a decision tree; the closeness of the two numbers is not a coincidence, because the source of the contribution in both setups is the same three keyword indicators. The recall column shows why the baseline has to be read: while the most frequent class gives an accuracy of 0.7017, it finds **none** of the suspicious ones. The bag of words passes the line by 0.0833 — more than twice what the five hand-counted numbers added. Even eight columns add 0.0283, which means the eight most frequent stems do about the same work as five hand-picked indicators. The contribution saturates at 64 columns: between 64 and 96 the held-out score stays **exactly the same** while the training score rises from 0.8333 to 0.8350. Adding columns always raises the training score; this is not a measurement but a requirement of the definition. #### Weighting and the Ceiling What is written into the cell of the bag is a separate decision: a presence indicator, a raw count, or the frequency weighted by its sparsity across documents. ```python run("count weighting", bag(RANKED, "count"), len(RANKED)) run("inverse document frequency", bag(RANKED, "weighted"), len(RANKED)) KEPT = [t for t in RANKED if t not in STOP_STEMS] run("stop words removed", bag(KEPT), len(KEPT)) KIND_LABEL = {} for name, _, _ in KINDS: p = [x["suspicious"] for x in TRAIN if x["kind"] == name] KIND_LABEL[name] = int(sum(p) * 2 > len(p)) ceiling = sum(KIND_LABEL[x["kind"]] == x["suspicious"] for x in HELD_OUT) / len(HELD_OUT) print(f"\n{'ceiling (event kind known)':<26} {'':>7} {'':>9} {ceiling:>9.4f} {'':>10}" f" {ceiling - BASELINE[0]:>+8.4f}") ``` ``` count weighting 96 0.8333 0.8217 0.5698 +0.0833 inverse document frequency 96 0.8150 0.8217 0.5363 +0.0833 stop words removed 90 0.8292 0.8150 0.5419 +0.0767 ceiling (event kind known) 0.8250 +0.0867 ``` Term weighting adds **nothing** in this corpus: all three rows give 0.8217. The reason lies in the text itself. The notes are short, and a stem almost always occurs once in a note, so a raw count cannot be told apart from an indicator column. Inverse document frequency weighting rescales the columns but does not break the ordering; while the accuracy stays the same, the recall falls from 0.5698 to 0.5363. The difference it adds is 0.0000. Dropping stop words **loses** 0.0067. Three of the six stop words are markers of particular patterns in this setup: `ile` (with) occurs only in the ordinary phrase, `ama` (but) only in the fault phrase. The stop word list is read from the task, not from the language; a general list throws away information here. The last row closes the remaining gap. A procedure that knows the event kind gives 0.8250, and the bag of words, at 0.8217, is 0.0033 below it. Since the label is generated from the kind alone, this is the highest number that can be read from the text: a better representation cannot win more than 0.0033 in this setup. The source of the difference can be written down as well — the model is the same on every row, the metric and the held-out set are the same, and **the difference comes from the representation**. #### Summary - The domain baseline is five hand-counted numbers and it gives 0.7383; the most frequent class stays at 0.7017 and finds none of the suspicious ones. - The bag of words of 96 stems gives 0.8217 and passes the line by 0.0833 — more than twice what the five hand-counted numbers added. Because the model and the metric are fixed, the whole difference comes from the representation. - Vocabulary size adds contribution from 8 to 64 and saturates there: while the difference between 64 and 96 is exactly zero, the training score rises from 0.8333 to 0.8350. - Term weighting adds 0.0000; because the notes are short, the count cannot be told apart from the indicator, and inverse document frequency weighting moves the recall rather than the accuracy (from 0.5698 to 0.5363). Dropping stop words loses 0.0067, because three words on the list are pattern markers in this setup. - The ceiling that knows the event kind is 0.8250; the bag is 0.0033 below the ceiling, and the remaining gap is not in the text. #### Next Step The bag of words turned a note into 96 columns, but the columns are entirely strangers to one another: the `muhur` (seal) and `kelepce` (clamp) columns are as far apart as the `muhur` and `` columns. All the model learns is the relation of each column to the label, and this relation is learned from scratch for every column. Yet in the notes `muhur` and `kelepce` occur next to similar words. The next lesson uses this: every stem is represented by a vector instead of a number, and the vector is learned from the **context**. The measure is whether the learned neighborhood turns out to be meaningful and in how many cases the vector arithmetic holds. ### Word Embeddings Source: https://academia.sh/en/courses/nlp-and-vision/word-embeddings Giving every item of the vocabulary a vector learned from context and counting neighborhood against the setup's own phrase pools: while the expected rate of a random neighbor is 0.2543, the raw co-occurrence vector gives 0.4938 and the eight-dimensional learned embedding gives 0.6049. The total number misleads — 49 of the 81 forms are rare nouns that occur in a single pattern, and there even raw counts give 0.7755, while on frequent forms random is 0.0462, raw counts 0.0625 and the learned embedding 0.2500. Analogy holds in 0.0972 of 72 tests, and the ones that do not hold land not on the most frequent word but on the forms with the narrowest context. The bag of words turned a note into 96 columns, but the columns were strangers to one another: the `muhur` (seal) column was as far from the `kelepce` (clamp) column as from the `` column. Yet in the notes these two occur in the same phrases, next to the same words. In this lesson every vocabulary item is given a vector, and the vector is learned not from the label but from the **context**. The claim is distributional meaning: words that occur in similar contexts get similar vectors. The embedding table itself was measured in the Deep Learning course, in `architectures/08`, and its procedure is not repeated. What is added here is moving the measure: the number is not the accuracy of a classifier but **the order of the vectors themselves**. Two tests are run, and both are read against their own baseline. - **NL17.** Tokenization here is the **second** decision of `nlp/01`: dropping punctuation, lowercasing, number folding. Stemming is not applied, because one of the things to be measured is whether inflectional suffixes leave a trace in the vectors. The vocabulary is 124 items. - **NL18.** The embedding table was measured in the Deep Learning course, `architectures/08`; the layer procedure is not repeated. - **NL19.** The model is written within the lesson and it **is the model**: a center vector and a context vector for every item, eight dimensions, 1,984 parameters in total. - **NL20.** Training: context window of two, two negative samples for every distinct pair, a weight of $\log(1+n)$ on the pair count, 40 epochs, step 0.08. The negative sample is drawn from the vocabulary with its own frequency. - **NL21.** The vectors are learned **from the training set only**; the pairs are extracted from 1,200 notes. - **NL22.** The metric is **cosine similarity**: the vectors are brought to unit length and the dot product is read. - **NL23.** The truth of the neighborhood test comes from the setup: a form belongs to the group of a phrase pool if its root occurs in that one pool only. 81 of the 124 items are like this. - **NL24.** The domain baseline is **the raw co-occurrence vector**: it learns nothing, it only counts. The absolute baseline is the random neighbor, and its expected rate is computed from the group sizes. #### Vectors Learned from Context The notes are regenerated from the same seed as the setup of `nlp/01`. ```python # COMMON — FICTIONAL. The field notes are generated within the lesson, seed 20260218; # they do not come from a source system. The vocabulary has 45 inflected roots and # 40 rare nouns. SEED, NOTE_COUNT, M32 = 20260218, 1800, 0xFFFFFFFF NOUNS = { # 0 base, 1 genitive, 2 locative, 3 possessive, 4 plural "sayac": "sayac sayacin sayacta sayaci sayaclar", "kapak": "kapak kapagin kapakta kapagi kapaklar", "muhur": "muhur muhurun muhurde muhuru muhurler", "vana": "vana vananin vanada vanasi vanalar", "kelepce": "kelepce kelepcenin kelepcede kelepcesi kelepceler", "boru": "boru borunun boruda borusu borular", "pano": "pano panonun panoda panosu panolar", "kapi": "kapi kapinin kapida kapisi kapilar", "zil": "zil zilin zilde zili ziller", "bahce": "bahce bahcenin bahcede bahcesi bahceler", "apartman": "apartman apartmanin apartmanda apartmani apartmanlar", "bodrum": "bodrum bodrumun bodrumda bodrumu bodrumlar", "giris": "giris girisin giriste girisi girisler", "adres": "adres adresin adreste adresi adresler", "okuma": "okuma okumanin okumada okumasi okumalar", "olcum": "olcum olcumun olcumde olcumu olcumler", "erisim": "erisim erisimin erisimde erisimi erisimler", "ziyaret": "ziyaret ziyaretin ziyarette ziyareti ziyaretler", "abone": "abone abonenin abonede abonesi aboneler", "arac": "arac aracin aracta araci araclar", "kir": "kir kirin kirde kiri kirler", "iz": "iz izin izde izi izler", "sokak": "sokak sokagin sokakta sokagi sokaklar", "ev": "ev evin evde evi evler", } ADJECTIVES = "kirik gevsek kirli normal bozuk acik kapali eksik temiz yeni yok".split() VERBS = { # 0 passive past, 1 negative "calis": "calisiyor calismiyor", "don": "donuyor donmuyor", "oku": "okundu okunmadi", "ac": "acildi acilmadi", "gir": "girildi girilemedi", "ulas": "ulasildi ulasilamadi", "tekrarla": "tekrarlandi tekrarlanmadi", "bul": "bulundu bulunmadi", "sagla": "saglandi saglanamadi", "degil": "degil degildi", } RARE_NOUNS = """kablo kablosu kabloda|cam cami camda|tel teli telde|mandal mandali mandalda merdiven merdiveni merdivende|tahta tahtasi tahtada|cukur cukuru cukurda|kar kari karda buz buzu buzda|cop copu copta|duvar duvari duvarda|kutu kutusu kutuda|vida vidasi vidada conta contasi contada|filtre filtresi filtrede|bant banti bantta|etiket etiketi etikette kilit kiliti kilitte|asansor asansoru asansorde|zemin zemini zeminde|tavan tavani tavanda kuyu kuyusu kuyuda|depo deposu depoda|ocak ocagi ocakta|direk diregi direkte kanal kanali kanalda|numara numarasi numarada|cati catisi catida|baca bacasi bacada kova kovasi kovada|levha levhasi levhada|somun somunu somunda|tapa tapasi tapada izolasyon izolasyonu izolasyonda|supap supabi supapta|menfez menfezi menfezde kayit kaydi kayitta|temel temeli temelde|kose kosesi kosede|koruma korumasi korumada""" PHRASES = { "ordinary": ["{sayac.0} {calis.0}", "{okuma.0} normal", "{olcum.0} {tekrarla.0}", "{sayac.1} {erisim.3} {sagla.0}", "{kapak.0} temiz", "{vana.0} acik", "{sayac.2} {iz.0} {bul.1}", "{olcum.0} icin {erisim.0} {sagla.0}", "{sayac.1} {kapak.3} yeni", "{muhur.0} ile {kapak.0} temiz"], "fault": ["{sayac.0} {don.1}", "{kapak.0} kirik", "{pano.0} bozuk", "{sayac.1} {kapak.3} kirik", "{olcum.0} {oku.1}", "{vana.0} bozuk", "{sayac.2} {kir.0} {bul.0}", "{sayac.4} {calis.1}", "{sayac.0} {calis.0} ama {okuma.0} yok", "{pano.1} {kapak.3} eksik"], "unreachable": ["{kapi.0} {ac.1}", "{zil.0} {calis.1}", "{abone.0} {ev.2} {degil.1}", "{erisim.0} {sagla.1}", "{bodrum.0} {kapi.3} kapali", "{apartman.1} {giris.3} kapali", "{adres.2} {ulas.1}", "{bahce.1} {kapi.3} kapali", "{apartman.2} {zil.0} {calis.1}", "{sayac.2} {erisim.0} yok"], "illicit": ["{muhur.0} kirik", "{kelepce.0} gevsek", "{muhur.2} {iz.0} {bul.0}", "{sayac.1} {muhur.3} kirik", "{kelepce.2} {iz.0} {bul.0}", "{muhur.0} eksik", "{vana.1} {kelepce.3} gevsek", "{muhur.1} {iz.3} {bul.0}", "{boru.2} {kelepce.0} eksik", "{kapak.2} bir {iz.0} {bul.0}"], } EXTRA_PHRASES = ["{ziyaret.0} {tekrarla.0}", "{sokak.2} {arac.0} {bul.0}", "{adres.0} yeni", "{bahce.1} {kapi.3} acik", "{apartman.2} {gir.0}", "{ev.1} {kapi.3} acik", "{kapak.0} kirli", "{olcum.0} {tekrarla.1}", "{arac.0} {sokak.2}", "bu {ziyaret.2} {olcum.0} {tekrarla.0}"] KINDS = [("ordinary", 0.36, 0.04), ("fault", 0.26, 0.28), ("unreachable", 0.16, 0.15), ("illicit", 0.22, 0.82)] FORMS = {k: v.split() for d in (NOUNS, VERBS) for k, v in d.items()} RARE = [s.split() for s in RARE_NOUNS.replace("\n", "|").split("|")] WEIGHTS = [1 / (i + 1) ** 1.6 for i in range(len(RARE))] WEIGHTS = [w / sum(WEIGHTS) for w in WEIGHTS] ROOT = {f: k for k, fs in FORMS.items() for f in fs} ROOT.update({f: fs[0] for fs in RARE for f in fs}) ROOT.update({w: w for w in ADJECTIVES}) STOP_WORDS = ["ve", "ile", "bir", "bu", "ama", "icin"] ROOT.update({w: w for w in STOP_WORDS}) def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def render_phrase(template): parts = [] for part in template.split(): if part.startswith("{"): name, _, i = part[1:-1].partition(".") parts.append(FORMS[name][int(i)]) else: parts.append(part) return " ".join(parts) DATA = [] for i in range(NOTE_COUNT): r = rng(SEED + i) kind = KINDS[pick(r(), [x[1] for x in KINDS])] clauses = [render_phrase(PHRASES[kind[0]][int(r() * 10)])] if r() < 0.55: clauses.append(render_phrase(EXTRA_PHRASES[int(r() * 10)])) if r() < 0.30: clauses.append(render_phrase(PHRASES[kind[0]][int(r() * 10)])) if r() < 0.22: forms, adjective = RARE[pick(r(), WEIGHTS)], ADJECTIVES[int(r() * len(ADJECTIVES))] clauses.append(forms[2] + " kir bulundu" if r() < 0.34 else forms[int(r() * 2)] + " " + adjective) note = clauses[0] for x in clauses[1:]: note += (" ve " if r() < 0.22 else ", ") + x if r() < 0.52: note += ", okuma " + str(240 + int(r() * 1760)) DATA.append({"id": 10001 + i, "note": note[0].upper() + note[1:] + ".", "kind": kind[0], "suspicious": int(r() < kind[2])}) TRAIN, HELD_OUT = DATA[:1200], DATA[1200:] print(f"{len(DATA)} fictional notes, {len(FORMS) + len(RARE) + len(ADJECTIVES)} roots, " f"suspicion rate {sum(x['suspicious'] for x in DATA) / len(DATA):.4f}") print(DATA[3]["note"]) ``` ``` 1800 fictional notes, 85 roots, suspicion rate 0.2956 Olcum icin erisim saglandi, sayacin kapagi yeni, mandalda kir bulundu. ``` Training sees every distinct context pair once: the vector of the center word is moved closer to the context vector of the true neighbor and away from those of the two negative samples drawn from the vocabulary. ```python import math from collections import Counter DIM, WINDOW, NEGATIVES, EPOCHS, STEP = 8, 2, 2, 40, 0.08 for x in DATA: # second decision of nlp/01: drop punctuation, lowercase, fold numbers x["tokens"] = [("" if t.isdigit() else t) for t in (s.strip(",.").lower() for s in x["note"].split()) if t] FREQ = Counter(t for x in TRAIN for t in x["tokens"]) VOCAB = sorted(FREQ, key=lambda t: (-FREQ[t], t)) POS = {t: i for i, t in enumerate(VOCAB)} PAIRS = Counter() for x in TRAIN: ids = [POS[t] for t in x["tokens"]] for i, a in enumerate(ids): for j in range(max(0, i - WINDOW), min(len(ids), i + WINDOW + 1)): if j != i: PAIRS[(a, ids[j])] += 1 T = sum(FREQ.values()) NOISE = [FREQ[t] / T for t in VOCAB] # the negative sample is drawn with the word's own frequency r = rng(SEED + 31000) U = [[(r() - 0.5) * 0.6 for _ in range(DIM)] for _ in VOCAB] C = [[0.0] * DIM for _ in VOCAB] for _ in range(EPOCHS): for (a, b), n in PAIRS.items(): w = math.log(1 + n) for k, target in [(b, 1.0)] + [(pick(r(), NOISE), 0.0) for _ in range(NEGATIVES)]: z = sum(U[a][i] * C[k][i] for i in range(DIM)) err = STEP * w * (1 / (1 + math.exp(-max(-30.0, min(30.0, z)))) - target) for i in range(DIM): U[a][i], C[k][i] = U[a][i] - err * C[k][i], C[k][i] - err * U[a][i] print(f"{len(VOCAB)} vocabulary items, {len(PAIRS)} distinct context pairs, " f"{2 * len(VOCAB) * DIM} parameters, {EPOCHS} epochs") def unit(v): u = math.sqrt(sum(x * x for x in v)) or 1.0 return [x / u for x in v] UNIT = [unit(v) for v in U] def cosine(i, j): return sum(UNIT[i][k] * UNIT[j][k] for k in range(DIM)) GROUP = {} for kind, templates in PHRASES.items(): for template in templates: for s in template.split(): name = s[1:-1].partition(".")[0] if s.startswith("{") else s GROUP.setdefault(name, set()).add(kind) for template in EXTRA_PHRASES: for s in template.split(): name = s[1:-1].partition(".")[0] if s.startswith("{") else s GROUP.setdefault(name, set()).add("extra") for f in RARE: GROUP[f[0]] = {"rare"} SINGLE = {name for name, v in GROUP.items() if len(v) == 1} FORM_GROUP = {t: list(GROUP[ROOT[t]])[0] for t in VOCAB if t in ROOT and ROOT[t] in SINGLE} GROUP_SIZE = Counter(FORM_GROUP.values()) print(f"{len(FORM_GROUP)} forms belong to a single phrase pool: " + ", ".join(f"{g} {n}" for g, n in sorted(GROUP_SIZE.items()))) ``` ``` 124 vocabulary items, 3216 distinct context pairs, 1984 parameters, 40 epochs 81 forms belong to a single phrase pool: extra 7, fault 7, illicit 6, ordinary 4, rare 49, unreachable 8 ``` #### Who Gets the Neighborhood For every form the nearest neighbor is found and the group of the neighbor is compared with the group of the form. The same test is applied to three representations: the expected rate of a random neighbor, the raw co-occurrence vector and the learned embedding. ```python COUNT_VEC = [[0.0] * len(VOCAB) for _ in VOCAB] for (a, b), n in PAIRS.items(): COUNT_VEC[a][b] = float(n) COUNT_UNIT = [unit(v) for v in COUNT_VEC] def neighbor(vecs, i): best, best_j = -2.0, i for j in range(len(VOCAB)): if j != i: d = sum(vecs[i][k] * vecs[j][k] for k in range(len(vecs[i]))) if d > best: best, best_j = d, j return VOCAB[best_j] def hit_rate(vecs, forms): return sum(FORM_GROUP.get(neighbor(vecs, POS[f])) == FORM_GROUP[f] for f in forms) / len(forms) def row(name, forms): expected = sum(GROUP_SIZE[FORM_GROUP[f]] - 1 for f in forms) / (len(forms) * (len(VOCAB) - 1)) print(f"{name:<24} {len(forms):>5} {expected:>9.4f} {hit_rate(COUNT_UNIT, forms):>10.4f}" f" {hit_rate(UNIT, forms):>10.4f}") ALL_FORMS = list(FORM_GROUP) print(f"\n{'neighborhood test':<24} {'forms':>5} {'random':>9} {'raw counts':>10}" f" {'learned':>10}") row("all", ALL_FORMS) row("frequent forms only", [f for f in ALL_FORMS if FORM_GROUP[f] != "rare"]) row("rare nouns only", [f for f in ALL_FORMS if FORM_GROUP[f] == "rare"]) ``` ``` neighborhood test forms random raw counts learned all 81 0.2543 0.4938 0.6049 frequent forms only 32 0.0462 0.0625 0.2500 rare nouns only 49 0.3902 0.7755 0.8367 ``` The total row gives the expected order: random 0.2543, raw counts 0.4938, learned embedding 0.6049. The learned representation adds 0.1111 over the counts, which learn nothing. Without reading the subrows this number is misleading: 49 of the 81 forms tested are rare nouns, and all of them occur in **a single phrase pattern**, which means they are one another's context. There the random expectation is 0.3902 and the raw counts give 0.7755. On frequent forms the table reverses. On the 32 frequent forms the random expectation is 0.0462, and the raw counts at 0.0625 are only 0.0163 above it — the baseline that learns nothing really knows nothing here. The learned embedding gives 0.2500, which is five times random and four times the raw counts. This is where learning adds something, and it does not show in the total row. This is exactly the second claim of the course: when a domain metric is reduced to a single number, which subgroup the number comes from is lost. #### Does Vector Arithmetic Hold The second test is harsher. Two roots that take the same inflectional suffix are chosen, and the question is asked with the vectors: if the difference that goes from the form `sayac` (meter) to the form `sayacin` (of the meter) is added to `muhur` (seal), which form comes out. The first vector is subtracted from the second, the third is added, the form closest to the result is searched for, and the correct answer is `muhurun` (of the seal). ```python ANALOGIES = [] for k in range(1, 5): pairs_k = [NOUNS[a].split() for a in NOUNS if NOUNS[a].split()[0] in VOCAB and NOUNS[a].split()[k] in VOCAB] for a in pairs_k: for b in pairs_k: if a[0] != b[0]: ANALOGIES.append((a[0], a[k], b[0], b[k])) def solve(vecs, a, b, c): v = [vecs[POS[b]][i] - vecs[POS[a]][i] + vecs[POS[c]][i] for i in range(len(vecs[0]))] best, best_j = -2.0, 0 for j in range(len(VOCAB)): if VOCAB[j] not in (a, b, c): d = sum(v[i] * vecs[j][i] for i in range(len(v))) if d > best: best, best_j = d, j return VOCAB[best_j] print(f"\n{'analogy':<18} {'tests':>6} {'hits':>7} {'same root':>9}" f" most given answer") for name, vecs in (("raw counts", COUNT_UNIT), ("learned embedding", UNIT)): hits = same_root = 0 answers = Counter() for a, b, c, y in ANALOGIES: t = solve(vecs, a, b, c) hits, same_root = hits + (t == y), same_root + (t != y and ROOT.get(t) == ROOT.get(c)) answers[t] += 1 n, (top, top_n) = len(ANALOGIES), answers.most_common(1)[0] print(f"{name:<18} {n:>6} {hits / n:>7.4f} {same_root / n:>9.4f} {top} {top_n / n:.4f}") ``` ``` analogy tests hits same root most given answer raw counts 72 0.1250 0.0000 kelepcede 0.1111 learned embedding 72 0.0972 0.0556 kelepcesi 0.0694 ``` Of the seventy-two tests, 0.1250 hold with the raw counts and 0.0972 with the learned embedding. Both are low, and this result is not hidden: at this scale analogy **does not hold**. For a suffix to leave a shared shift in the vectors, the same suffix has to be seen on many roots and in contexts that are separate from one another. In this setup every root and suffix pair occurs in one or two patterns; the vector carries the context of the root, not the context of the suffix. Of the seventy-two tests, 0.0556 land on another form of the same root, which means the root is captured and the suffix is not. The expected flaw does not appear. The three most frequent forms are `okuma` (reading), `` and `olcum` (measurement), and none of them comes out as an answer; the answers gather on the forms with the narrowest context — `kelepcede` (in the clamp) with 0.1111 with the raw counts, `kelepcesi` (its clamp) with 0.0694 with the embedding. The same representation beats the baseline in the neighborhood test and fails to in analogy; the two metrics read the same vectors in two different orders. #### Summary - An eight-dimensional embedding with 1,984 parameters was trained for 40 epochs on the 3,216 distinct context pairs from 1,200 notes; the vectors were learned from the context, not from the label. - In the neighborhood test the random expectation gave 0.2543, the raw co-occurrence 0.4938, the learned embedding 0.6049; what the learned representation adds over the counts is 0.1111. - The total number misleads: 49 of the 81 forms are rare nouns that occur in a single pattern, and there even the raw counts give 0.7755. On the 32 frequent forms random gives 0.0462, raw counts 0.0625, the learned embedding 0.2500 — this is where learning adds something, and it does not show in the total row. - Analogy does not hold: of 72 tests 0.1250 are right with the raw counts and 0.0972 with the embedding. For the suffix to leave a trace in the vector, the same suffix has to be seen in many separate contexts. - Of the seventy-two tests, 0.0556 land on another form of the same root; the answers gather not on the most frequent word but on the forms with the narrowest context (`kelepcesi` 0.0694). #### Next Step Over three lessons a note was reduced to a single line: first the token list, then the 96-column bag, then one vector per word. The decision given was every time about **the whole document** — is this note suspicious. Yet what each token inside the note reports was never asked: in the note `sayacin kapagi kirik` (the meter's cover is broken), which token names the hardware, which the location, which the state. The next lesson takes the label from the document and gives it **to every token separately**. Then what accuracy counts changes as well: per token, or per entity that spans several tokens. ### Sequence Labeling Source: https://academia.sh/en/courses/nlp-and-vision/sequence-labeling Taking the label from the document and giving it to every token separately, and measuring what accuracy counts: writing the most frequent label to every token, the domain baseline, gives a token accuracy of 0.5061 but an entity-level F1 of 0.0000 — not a single entity is found. The procedure that looks at the last three characters gives the same token accuracy of 0.9824 under both labeling schemes while it gives 0.9267 and 0.9600 at the entity level; the procedure that memorizes the token has a token accuracy of 1.0000 and stays at 0.9665 in the type scheme. In part of speech the baseline of the same form gives 0.5473 and does not mislead there. Over three lessons a note was reduced to a single line, and the decision given was every time about the whole document. Which token names what inside the note was never asked. In the note `sayacin kapagi kirik` (the meter's cover is broken) the first two tokens name a piece of hardware and the third reports its state; yet in the note `boruda kelepce eksik` (the clamp is missing on the pipe) the first two tokens are **two separate** pieces of hardware. This lesson distributes the label to the tokens. When the label moves, the metric changes too: at the token level accuracy counts every token separately, while at the entity level an entity is counted only when it holds its start, its end and its type together. The domain baseline here is **writing the most frequent label to every token**, and at the token level it comes out surprisingly high. - **NL25.** Tokenization is the decision of `nlp/03`: dropping punctuation, lowercasing, number folding. Stemming is not applied, because part of the boundary decision is read from the suffix. - **NL26.** The entity types are **fiction** and are derived from the fiction's own dictionary: hardware has seven roots, location nine roots, measurement two roots plus the number token. The other tokens are outside any entity. - **NL27.** The boundary rule: if a noun in the base or genitive case is followed by a noun in the possessive case, the two are **a single entity**; a number token attaches to the preceding measurement token. In all other cases every token is its own entity. - **NL28.** The part-of-speech label is also read from the dictionary: noun, adjective, verb, stop word, number. - **NL29.** The domain baseline is **writing the most frequent label to every token**, in both tasks. - **NL30.** The model is a memorization table: for every key the most frequent label in the training set. Two keys are tried — the token itself and its last three characters. - **NL31.** There are two labeling schemes: type only, and type with a begin/inside distinction (B/I). Token-level accuracy is counted in both **over the type**, so that the two schemes are compared with the same metric. - **NL32.** The entity-level metric is exact match. The definitions of precision, recall and F1 were built in the Supervised Learning course and are not repeated; the only thing added here is **what is counted**. #### Entities and the Two Labeling Schemes The notes are generated from the same seed as the setup of `nlp/01`. ```python # COMMON — FICTIONAL. The field notes are generated within the lesson, seed 20260218; # they do not come from a source system. The vocabulary has 45 inflected roots and # 40 rare nouns. SEED, NOTE_COUNT, M32 = 20260218, 1800, 0xFFFFFFFF NOUNS = { # 0 base, 1 genitive, 2 locative, 3 possessive, 4 plural "sayac": "sayac sayacin sayacta sayaci sayaclar", "kapak": "kapak kapagin kapakta kapagi kapaklar", "muhur": "muhur muhurun muhurde muhuru muhurler", "vana": "vana vananin vanada vanasi vanalar", "kelepce": "kelepce kelepcenin kelepcede kelepcesi kelepceler", "boru": "boru borunun boruda borusu borular", "pano": "pano panonun panoda panosu panolar", "kapi": "kapi kapinin kapida kapisi kapilar", "zil": "zil zilin zilde zili ziller", "bahce": "bahce bahcenin bahcede bahcesi bahceler", "apartman": "apartman apartmanin apartmanda apartmani apartmanlar", "bodrum": "bodrum bodrumun bodrumda bodrumu bodrumlar", "giris": "giris girisin giriste girisi girisler", "adres": "adres adresin adreste adresi adresler", "okuma": "okuma okumanin okumada okumasi okumalar", "olcum": "olcum olcumun olcumde olcumu olcumler", "erisim": "erisim erisimin erisimde erisimi erisimler", "ziyaret": "ziyaret ziyaretin ziyarette ziyareti ziyaretler", "abone": "abone abonenin abonede abonesi aboneler", "arac": "arac aracin aracta araci araclar", "kir": "kir kirin kirde kiri kirler", "iz": "iz izin izde izi izler", "sokak": "sokak sokagin sokakta sokagi sokaklar", "ev": "ev evin evde evi evler", } ADJECTIVES = "kirik gevsek kirli normal bozuk acik kapali eksik temiz yeni yok".split() VERBS = { # 0 passive past, 1 negative "calis": "calisiyor calismiyor", "don": "donuyor donmuyor", "oku": "okundu okunmadi", "ac": "acildi acilmadi", "gir": "girildi girilemedi", "ulas": "ulasildi ulasilamadi", "tekrarla": "tekrarlandi tekrarlanmadi", "bul": "bulundu bulunmadi", "sagla": "saglandi saglanamadi", "degil": "degil degildi", } RARE_NOUNS = """kablo kablosu kabloda|cam cami camda|tel teli telde|mandal mandali mandalda merdiven merdiveni merdivende|tahta tahtasi tahtada|cukur cukuru cukurda|kar kari karda buz buzu buzda|cop copu copta|duvar duvari duvarda|kutu kutusu kutuda|vida vidasi vidada conta contasi contada|filtre filtresi filtrede|bant banti bantta|etiket etiketi etikette kilit kiliti kilitte|asansor asansoru asansorde|zemin zemini zeminde|tavan tavani tavanda kuyu kuyusu kuyuda|depo deposu depoda|ocak ocagi ocakta|direk diregi direkte kanal kanali kanalda|numara numarasi numarada|cati catisi catida|baca bacasi bacada kova kovasi kovada|levha levhasi levhada|somun somunu somunda|tapa tapasi tapada izolasyon izolasyonu izolasyonda|supap supabi supapta|menfez menfezi menfezde kayit kaydi kayitta|temel temeli temelde|kose kosesi kosede|koruma korumasi korumada""" PHRASES = { "ordinary": ["{sayac.0} {calis.0}", "{okuma.0} normal", "{olcum.0} {tekrarla.0}", "{sayac.1} {erisim.3} {sagla.0}", "{kapak.0} temiz", "{vana.0} acik", "{sayac.2} {iz.0} {bul.1}", "{olcum.0} icin {erisim.0} {sagla.0}", "{sayac.1} {kapak.3} yeni", "{muhur.0} ile {kapak.0} temiz"], "fault": ["{sayac.0} {don.1}", "{kapak.0} kirik", "{pano.0} bozuk", "{sayac.1} {kapak.3} kirik", "{olcum.0} {oku.1}", "{vana.0} bozuk", "{sayac.2} {kir.0} {bul.0}", "{sayac.4} {calis.1}", "{sayac.0} {calis.0} ama {okuma.0} yok", "{pano.1} {kapak.3} eksik"], "unreachable": ["{kapi.0} {ac.1}", "{zil.0} {calis.1}", "{abone.0} {ev.2} {degil.1}", "{erisim.0} {sagla.1}", "{bodrum.0} {kapi.3} kapali", "{apartman.1} {giris.3} kapali", "{adres.2} {ulas.1}", "{bahce.1} {kapi.3} kapali", "{apartman.2} {zil.0} {calis.1}", "{sayac.2} {erisim.0} yok"], "illicit": ["{muhur.0} kirik", "{kelepce.0} gevsek", "{muhur.2} {iz.0} {bul.0}", "{sayac.1} {muhur.3} kirik", "{kelepce.2} {iz.0} {bul.0}", "{muhur.0} eksik", "{vana.1} {kelepce.3} gevsek", "{muhur.1} {iz.3} {bul.0}", "{boru.2} {kelepce.0} eksik", "{kapak.2} bir {iz.0} {bul.0}"], } EXTRA_PHRASES = ["{ziyaret.0} {tekrarla.0}", "{sokak.2} {arac.0} {bul.0}", "{adres.0} yeni", "{bahce.1} {kapi.3} acik", "{apartman.2} {gir.0}", "{ev.1} {kapi.3} acik", "{kapak.0} kirli", "{olcum.0} {tekrarla.1}", "{arac.0} {sokak.2}", "bu {ziyaret.2} {olcum.0} {tekrarla.0}"] KINDS = [("ordinary", 0.36, 0.04), ("fault", 0.26, 0.28), ("unreachable", 0.16, 0.15), ("illicit", 0.22, 0.82)] FORMS = {k: v.split() for d in (NOUNS, VERBS) for k, v in d.items()} RARE = [s.split() for s in RARE_NOUNS.replace("\n", "|").split("|")] WEIGHTS = [1 / (i + 1) ** 1.6 for i in range(len(RARE))] WEIGHTS = [w / sum(WEIGHTS) for w in WEIGHTS] ROOT = {f: k for k, fs in FORMS.items() for f in fs} ROOT.update({f: fs[0] for fs in RARE for f in fs}) ROOT.update({w: w for w in ADJECTIVES}) STOP_WORDS = ["ve", "ile", "bir", "bu", "ama", "icin"] ROOT.update({w: w for w in STOP_WORDS}) def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def render_phrase(template): parts = [] for part in template.split(): if part.startswith("{"): name, _, i = part[1:-1].partition(".") parts.append(FORMS[name][int(i)]) else: parts.append(part) return " ".join(parts) DATA = [] for i in range(NOTE_COUNT): r = rng(SEED + i) kind = KINDS[pick(r(), [x[1] for x in KINDS])] clauses = [render_phrase(PHRASES[kind[0]][int(r() * 10)])] if r() < 0.55: clauses.append(render_phrase(EXTRA_PHRASES[int(r() * 10)])) if r() < 0.30: clauses.append(render_phrase(PHRASES[kind[0]][int(r() * 10)])) if r() < 0.22: forms, adjective = RARE[pick(r(), WEIGHTS)], ADJECTIVES[int(r() * len(ADJECTIVES))] clauses.append(forms[2] + " kir bulundu" if r() < 0.34 else forms[int(r() * 2)] + " " + adjective) note = clauses[0] for x in clauses[1:]: note += (" ve " if r() < 0.22 else ", ") + x if r() < 0.52: note += ", okuma " + str(240 + int(r() * 1760)) DATA.append({"id": 10001 + i, "note": note[0].upper() + note[1:] + ".", "kind": kind[0], "suspicious": int(r() < kind[2])}) TRAIN, HELD_OUT = DATA[:1200], DATA[1200:] print(f"{len(DATA)} fictional notes, {len(FORMS) + len(RARE) + len(ADJECTIVES)} roots, " f"suspicion rate {sum(x['suspicious'] for x in DATA) / len(DATA):.4f}") print(DATA[3]["note"]) ``` ``` 1800 fictional notes, 85 roots, suspicion rate 0.2956 Olcum icin erisim saglandi, sayacin kapagi yeni, mandalda kir bulundu. ``` Entities and token labels are derived from the fiction's dictionary, with the same boundary rule in both schemes. The prediction procedure is the same in both: the most frequent label per key. ```python from collections import Counter ENTITY_TYPES = {"hardware": "sayac kapak muhur vana kelepce boru pano", "location": "kapi zil bahce apartman bodrum giris adres ev sokak", "measurement": "okuma olcum"} ROOT_TYPE = {k: t for t, v in ENTITY_TYPES.items() for k in v.split()} CASE = {f: i for v in NOUNS.values() for i, f in enumerate(v.split())} POS_TAG = {f: "noun" for v in list(NOUNS.values()) + [" ".join(s) for s in RARE] for f in v.split()} POS_TAG.update({f: "adjective" for f in ADJECTIVES}) POS_TAG.update({f: "verb" for v in VERBS.values() for f in v.split()}) POS_TAG.update({f: "stop word" for f in STOP_WORDS}, **{"": "number"}) for x in DATA: x["tokens"] = [("" if t.isdigit() else t) for t in (s.strip(",.").lower() for s in x["note"].split()) if t] def entity_type(t): return "measurement" if t == "" else ROOT_TYPE.get(ROOT.get(t)) def attaches(a, b): # genitive or base + possessive is a single entity; a number attaches to the reading return b == "" or (CASE.get(a) in (0, 1) and CASE.get(b) == 3) def entities(tokens): found, i = [], 0 while i < len(tokens): if entity_type(tokens[i]) is None: i += 1 continue j = i + 1 while j < len(tokens) and entity_type(tokens[j]) and attaches(tokens[j - 1], tokens[j]): j += 1 found.append((i, j, entity_type(tokens[j - 1]))) i = j return found def label(tokens, begin_inside): out = ["outside"] * len(tokens) for i, j, kind in entities(tokens): for k in range(i, j): out[k] = (("B-" if k == i else "I-") + kind) if begin_inside else kind return out def decode(tokens, predicted, begin_inside): found, i = [], 0 while i < len(tokens): t = predicted[i] if t == "outside": i += 1 continue kind = t[2:] if begin_inside else t j = i + 1 while j < len(tokens) and predicted[j] == ("I-" + kind if begin_inside else kind): j += 1 found.append((i, j, kind)) i = j return found def build_table(begin_inside, key): counts = {} for x in TRAIN: for t, e in zip(x["tokens"], label(x["tokens"], begin_inside)): counts.setdefault(key(t), Counter())[e] += 1 default = Counter(e for x in TRAIN for e in label(x["tokens"], begin_inside)).most_common(1)[0][0] return {t: c.most_common(1)[0][0] for t, c in counts.items()}, default def evaluate(name, begin_inside, key=lambda t: t, constant=None): table, default = build_table(begin_inside, key) hit = total = tp = predicted_n = gold_n = 0 for x in HELD_OUT: tokens = x["tokens"] predicted = ([constant] * len(tokens) if constant else [table.get(key(t), default) for t in tokens]) kinds = [e[2:] if begin_inside and e != "outside" else e for e in predicted] hit += sum(a == b for a, b in zip(label(tokens, False), kinds)) total += len(tokens) found, gold = decode(tokens, predicted, begin_inside), entities(tokens) tp += sum(v in gold for v in found) predicted_n += len(found) gold_n += len(gold) precision, recall = tp / predicted_n if predicted_n else 0.0, tp / gold_n f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0 print(f"{name:<28} {hit / total:>9.4f} {precision:>9.4f} {recall:>11.4f} {f1:>7.4f}") MOST_COMMON = Counter(e for x in TRAIN for e in label(x["tokens"], False)).most_common(1)[0][0] print(f"{len(DATA)} notes contain {sum(len(entities(x['tokens'])) for x in DATA)} entities, " f"{sum(j - i > 1 for x in DATA for i, j, _ in entities(x['tokens']))} of them multi-token; " f"the most frequent token label is '{MOST_COMMON}'") print(f"\n{'named entity':<28} {'token':>9} {'precision':>9} {'recall':>11}" f" {'F1':>7}") LAST3 = (lambda t: t[-3:]) evaluate("most frequent label", False, constant=MOST_COMMON) evaluate("last 3 chars, type", False, LAST3) evaluate("last 3 chars, B/I", True, LAST3) evaluate("token identity, type", False) evaluate("token identity, B/I", True) ``` ``` 1800 notes contain 4378 entities, 1583 of them multi-token; the most frequent token label is 'outside' named entity token precision recall F1 most frequent label 0.5061 0.0000 0.0000 0.0000 last 3 chars, type 0.9824 0.9191 0.9343 0.9267 last 3 chars, B/I 0.9824 0.9421 0.9786 0.9600 token identity, type 1.0000 0.9774 0.9558 0.9665 token identity, B/I 1.0000 1.0000 1.0000 1.0000 ``` The set contains 4,378 entities and 1,583 of them span more than one token. The most frequent token label is outside any entity, and writing it to every token gives an accuracy of 0.5061 on the held-out set: more than half of the tokens are labeled correctly. The entity-level number for the same output is 0.0000, because not a single entity is found. The rule of the course offers no sharper example than this — read alone, 0.5061 makes one think the procedure has gone half the way, yet it has found nothing. The middle two rows show the real split. The procedure that looks at the last three characters gives the **same** token accuracy under both schemes: 0.9824. At the entity level it gives 0.9267 and 0.9600. The two schemes make the same token decisions; the only place they part is how they read the boundary, and the token-level metric **cannot see** this difference. The same output has two numbers, and which one gets written depends on the definition of the metric. How a token error turns into an entity error can also be read in the table: a token-level error of 0.0176 gives an F1 loss of 0.0733 in the type scheme. An entity counts only if all of its tokens are right, which means the error is multiplied along the entity. The last pair of rows shows this at the extreme: memorization gives a token accuracy of 1.0000 and stays at 0.9665 in the type scheme. Even with perfect token decisions there is a loss, because the type scheme counts two adjacent entities of the same type as a single entity — in the note `boruda kelepce eksik` two separate pieces of hardware collapse into one span. The begin/inside distinction carries this into the label and gives 1.0000. #### The Same Baseline, Another Task The second task is part of speech and has no entity level: the label is one-to-one with the token. ```python POS_COUNTS = Counter(POS_TAG[t] for x in HELD_OUT for t in x["tokens"]) POS_DEFAULT = Counter(POS_TAG[t] for x in TRAIN for t in x["tokens"]).most_common(1)[0][0] def measure_pos(name, key=None): table = {} if key: counts = {} for x in TRAIN: for t in x["tokens"]: counts.setdefault(key(t), Counter())[POS_TAG[t]] += 1 table = {k: v.most_common(1)[0][0] for k, v in counts.items()} hit = total = 0 for x in HELD_OUT: for t in x["tokens"]: hit += (table.get(key(t), POS_DEFAULT) if key else POS_DEFAULT) == POS_TAG[t] total += 1 print(f"{name:<28} {hit / total:>9.4f}") print("token types in the held-out set: " + ", ".join(f"{k} {v}" for k, v in POS_COUNTS.most_common())) print(f"\n{'part of speech':<28} {'token':>9}") measure_pos(f"most frequent label ({POS_DEFAULT})") measure_pos("last 3 chars", LAST3) measure_pos("token identity", lambda t: t) ``` ``` token types in the held-out set: noun 2181, adjective 622, verb 603, number 312, stop word 267 part of speech token most frequent label (noun) 0.5473 last 3 chars 0.9905 token identity 1.0000 ``` Writing the most frequent label to every token gives 0.5473 here, which is even higher than for the named entity. Of the 3,985 tokens of the held-out set 2,181 are nouns, and this share alone sets the baseline. The last three characters give 0.9905 and the token identity gives 1.0000. The numbers are high because the task is easy in this setup: the part of speech of a surface form does not change. The real reading is in comparing the two tasks. The baseline of the same form carries two different meanings in two tasks: in part of speech 0.5473 honestly reports the difficulty of the task, while in the named entity 0.5061 shows a procedure that has found nothing as halfway there. The difference is not in the procedure but in the metric — in one the labeled unit and the counted unit are the same, in the other they are not. #### Summary - The label was taken from the document and distributed to the tokens; the 1,800 notes hold 4,378 entities and 1,583 of them span more than one token. - The domain baseline is writing the most frequent label to every token: 0.5061 at the token level, F1 0.0000 at the entity level. Read alone, the token-level number makes one think the procedure has gone half the way. - The procedure that looks at the last three characters gives the same token accuracy (0.9824) under both labeling schemes while it gives 0.9267 and 0.9600 at the entity level; the difference lies only in how the boundary is read, and the token-level metric cannot see it. - A token-level error of 0.0176 gives an F1 loss of 0.0733 in the type scheme; memorization stays at 0.9665 in the type scheme with a token accuracy of 1.0000, because two adjacent entities of the same type collapse into one span. - In part of speech the baseline of the same form gives 0.5473 and does not mislead: there the labeled unit and the counted unit are the same. #### Next Step Over four lessons the text was turned into numbers: it was split into tokens, opened into columns, embedded into vectors and, in the last lesson, the label was distributed to the tokens. The decision given to the whole document was asked in this series only once, with a single binary question, and what was measured there was not the decision itself but **the representation**: what each representation adds. The event kind the note reports has four classes, however, and that decision was never made. The pipeline itself was never taken apart either — tokenization, weighting and classifier were measured together, and how many points each step adds was not asked separately. The next lesson classifies the document as a whole and counts the share of every step of the pipeline one by one. ### Text Classification Source: https://academia.sh/en/courses/nlp-and-vision/text-classification Giving a single event kind decision to the whole of a field note and counting separately how many points each of the three steps of the pipeline adds: the most frequent class baseline is 0.4420, seven hand-counted measures give 0.5500, the bag of words with a linear model gives 0.9620. Measured over twelve setups, the representation step adds 0.3787 on average, the classifier step 0.0500, term weighting 0.0140; both tokenization decisions give a negative contribution, and suffix clipping costs 0.0165 on average while bringing the vocabulary down from 109 to 89. The contribution of a step is not a single number: weighting rises to 0.0340 with the class mean and falls to minus 0.0060 with the linear model. Over four lessons the text was turned into numbers. How the tokenization decision moved the size of the vocabulary, what the bag of words and term weighting added to hand-counted features, how the embedding is learned from context and how the labels are distributed to **tokens** were measured. In all these measurements the decision unit was the token: every token received its own label, and the metric was read at the token or the entity level. Giving a single decision to the **whole** of a document was never tried. A field note is read and one of four event kinds is written: meter fault, suspected illicit use, unreachable, ordinary. In this setup the output is a single label and the length of the input is variable. The question of this lesson is this: **which step of the pipeline that produces this decision adds how many points**. There are three steps — tokenization, term weighting, classifier — and the contribution of each is not independent of the others. - **NL41.** The field notes are **fiction**. They are an extended version of the note field produced by `feature-selection/02` of the Data Preparation and Feature Engineering course: every note is built from an event phrase, sometimes a side record borrowed from another kind, a noise phrase and a line marker. Seed 20260218. - **NL42.** The event kind label is **the kind the note was generated from**; it is not derived from the words inside the note. Because of the borrowed side record the text sometimes points to another kind; this pulls the ceiling of the measure down and is a deliberate part of the fiction. - **NL43.** The set is 1,600 notes; the first 1,100 are the training set and the last 500 the held-out set. - **NL44.** **The domain baseline is the most frequent class.** In a four-class task random assignment stays around 0.25; the most frequent class is higher than that, and since the classes are imbalanced it is the real comparison. - **NL45.** **Every step of the pipeline is learned from the training set only**: the vocabulary, the document frequencies and the model weights. This rule was built in `feature-selection/05` of the Data Preparation and Feature Engineering course and continues here. - **NL46.** The vocabulary takes the words that occur in **at least two notes** in the training set. - **NL47.** Both classifiers are written within the lesson and are **the model**: the class mean (cosine similarity to the mean vector of every class) and a four-class linear model. #### A Single Decision for the Whole Document The input of classification is a document and its output is a label. The difference from sequence labeling is not a difference of scale but a difference of decision unit: there was one output for every token there, here there is one output for the whole note, and how many tokens the note consists of does not change the shape of the output. ```python # FICTIONAL field notes — an extended version of the note field of the Data Preparation and # Feature Engineering course. Seed 20260218. import math SEED, M32 = 20260218, 0xFFFFFFFF KINDS = ("ordinary", "unreachable", "meter fault", "illicit use") SHARES = (0.46, 0.22, 0.19, 0.13) PHRASES = {k: s.split("|") for k, s in zip(KINDS, ( "okuma normal|okuma normaldi|sayac erisimi saglandi|sayaca erisildi|olcum tekrarlandi|" "olcumu yineledik|vana kontrol edildi|vanada sorun yok|kapak temiz|kapakta iz yok|" "gosterge okunakli|gostergede rakam net", "abone evde degildi|abone bulunamadi|erisim saglanamadi|erisim yok|kapi zili calismiyor|" "zil calismadi|bahce kapisi kilitli|bahce kapisi kapali|bodrum katina inilemedi|" "bodrum kilitli|adres bulunamadi|adres eksik", "gosterge donmuyor|gostergede hareket yok|sayac cami kirik|sayacin cami catlak|" "rakam okunmuyor|rakamlar silik|ibre sabit kalmis|ibre kilitli|mekanizma takilmis|" "mekanizmada takilma|sayac durmus|sayac calismiyor", "muhur izi var|muhurde iz goruldu|muhur kopmus|muhurler kopuk|baglanti degistirilmis|" "baglantida degisiklik|kelepce gevsek|kelepcede gevseme|boru ek yeri yeni|boruda yeni ek|" "sayac ters takilmis|sayaci ters takmislar"))} NOISE = ("arac park halinde|kapak kirli|hava yagisli|adres teyit edildi|not kisa tutuldu|" "ikinci ziyaret|bahce icinde|kaldirim uzerinde").split("|") REGIONS = "kuzey hatti|guney hatti|dogu hatti|bati hatti|merkez hatti".split("|") SUSPICION_P = {"illicit use": 0.86, "meter fault": 0.21, "unreachable": 0.07, "ordinary": 0.03} def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def build_note(n): r = rng(SEED + 41000 + n) u, k = r(), 0.0 for i, p in enumerate(SHARES): k += p if u < k: t = KINDS[i] break p = [PHRASES[t][int(r() * 12)]] if r() < 0.55: p.append(PHRASES[t][int(r() * 12)]) if r() < 0.34: # side record borrowed from another kind p.append(PHRASES[KINDS[int(r() * 4)]][int(r() * 12)]) if r() < 0.62: p.append(NOISE[int(r() * 8)]) if r() < 0.30: p.append(REGIONS[int(r() * 5)]) return {"text": ", ".join(p), "kind": t, "suspicious": 1 if r() < SUSPICION_P[t] else 0} NOTES = [build_note(i) for i in range(1600)] TRAIN, HELD_OUT = NOTES[:1100], NOTES[1100:] KIND_INDEX = {t: i for i, t in enumerate(KINDS)} counts = {} for x in TRAIN: counts[x["kind"]] = counts.get(x["kind"], 0) + 1 MOST_COMMON = max(counts, key=counts.get) BASELINE = sum(x["kind"] == MOST_COMMON for x in HELD_OUT) / len(HELD_OUT) for i in (3, 25, 40): print(f"note: {NOTES[i]['text']} -> {NOTES[i]['kind']}") print(f"train {len(TRAIN)}, held-out {len(HELD_OUT)}, mean tokens " f"{sum(len(x['text'].split()) for x in NOTES) / len(NOTES):.2f}") print("in the held-out set " + ", ".join( f"{t} {sum(x['kind'] == t for x in HELD_OUT) / len(HELD_OUT):.4f}" for t in KINDS)) print(f"most frequent class '{MOST_COMMON}', domain baseline {BASELINE:.4f}") ``` ``` note: baglanti degistirilmis, kelepce gevsek, dogu hatti -> illicit use note: bodrum katina inilemedi, abone evde degildi, sayac cami kirik, bahce icinde -> unreachable note: vanada sorun yok, gostergede rakam net, sayac durmus, kuzey hatti -> ordinary train 1100, held-out 500, mean tokens 6.68 in the held-out set ordinary 0.4420, unreachable 0.2340, meter fault 0.1920, illicit use 0.1320 most frequent class 'ordinary', domain baseline 0.4420 ``` The second and third example notes show what the fiction makes difficult. The second was generated from the "unreachable" kind but carries a fault record (`sayac cami kirik`, the meter glass is broken); the third was generated from the "ordinary" kind but the phrase "sayac durmus" (the meter has stopped) comes from the fault phrases. The text does not fully determine the label, it only points to it. The domain baseline is 0.4420. This number is the rate in the held-out set of a procedure that writes "ordinary" on every note without looking at any word, and every number from here on is read against it. #### Three Steps, Twelve Setups The pipeline is the product of three decisions. Tokenization has three options: **coarse** (splitting on whitespace only, punctuation stays attached to the word), **careful** (comma stripped, lowercased), **clipped** (on top of careful, every word is cut down to its first five characters — the crudest approximation of stemming in an agglutinative language). Weighting has two options: raw **frequency**, or the product of the frequency with the **inverse document frequency**. The classifier has two options. The three decisions have twelve combinations, and all twelve are measured on the same held-out set. ```python # The three steps of the pipeline. Every step is learned from the TRAINING set ONLY. SPLIT = {"coarse": lambda m: m.split(), "careful": lambda m: [w.strip(",").lower() for w in m.split() if w.strip(",")], "clipped": lambda m: [w.strip(",").lower()[:5] for w in m.split() if w.strip(",")]} def build_vocabulary(D, split): d = {} for x in D: for w in set(split(x["text"])): d[w] = d.get(w, 0) + 1 words = sorted(w for w, c in d.items() if c >= 2) return ({w: i for i, w in enumerate(words)}, {i: math.log(len(D) / d[w]) for i, w in enumerate(words)}) def vectorize(text, split, vocab, idf=None): v = {} for w in split(text): if w in vocab: v[vocab[w]] = v.get(vocab[w], 0.0) + 1.0 if idf: for i in v: v[i] *= idf[i] n = math.sqrt(sum(u * u for u in v.values())) or 1.0 return {i: u / n for i, u in v.items()} def class_mean(D, V): # THE MODEL: the mean vector of every class C = [{} for _ in KINDS] for x, v in zip(D, V): c = C[KIND_INDEX[x["kind"]]] for i, u in v.items(): c[i] = c.get(i, 0.0) + u for c in C: n = math.sqrt(sum(u * u for u in c.values())) or 1.0 for i in c: c[i] /= n return lambda v: max(range(4), key=lambda j: sum(u * C[j].get(i, 0.0) for i, u in v.items())) def linear_model(D, V, size, epochs=150, step=3.0): # THE MODEL: four-class linear W, b = [[0.0] * size for _ in KINDS], [0.0] * 4 y, n = [KIND_INDEX[x["kind"]] for x in D], len(D) for _ in range(epochs): gW, gb = [{} for _ in KINDS], [0.0] * 4 for v, t in zip(V, y): o = [sum(W[j][i] * u for i, u in v.items()) + b[j] for j in range(4)] e = [math.exp(z - max(o)) for z in o] s = sum(e) for j in range(4): d = e[j] / s - (1.0 if j == t else 0.0) gb[j] += d for i, u in v.items(): gW[j][i] = gW[j].get(i, 0.0) + d * u for j in range(4): b[j] -= step * gb[j] / n for i, g in gW[j].items(): W[j][i] -= step * g / n return lambda v: max(range(4), key=lambda j: sum(W[j][i] * u for i, u in v.items()) + b[j]) def score(predict, D, V): return sum(predict(v) == KIND_INDEX[x["kind"]] for x, v in zip(D, V)) / len(D) KEYWORDS = ("muhur", "erisim", "sayac", "gosterge", "kelepce") def by_hand(text): # the hand-counted text measures of the Data Preparation and Feature Engineering course w = SPLIT["careful"](text) v = {0: len(text) / 60.0, 1: len(w) / 10.0} for j, a in enumerate(KEYWORDS): v[2 + j] = 1.0 if any(a in u for u in w) else 0.0 n = math.sqrt(sum(u * u for u in v.values())) or 1.0 return {i: u / n for i, u in v.items()} V_TRAIN, V_HELD = [by_hand(x["text"]) for x in TRAIN], [by_hand(x["text"]) for x in HELD_OUT] BY_HAND_SCORE = score(linear_model(TRAIN, V_TRAIN, 7), HELD_OUT, V_HELD) GRID = {} print(f"{'tokenization':<13}{'vocab':>7}{'weighting':>11}{'class mean':>12}{'linear':>10}") for name, split in SPLIT.items(): vocab, idf = build_vocabulary(TRAIN, split) for weighting, w in (("count", None), ("weighted", idf)): V_TRAIN = [vectorize(x["text"], split, vocab, w) for x in TRAIN] V_HELD = [vectorize(x["text"], split, vocab, w) for x in HELD_OUT] a = score(class_mean(TRAIN, V_TRAIN), HELD_OUT, V_HELD) d = score(linear_model(TRAIN, V_TRAIN, len(vocab)), HELD_OUT, V_HELD) GRID[(name, weighting)] = (a, d) print(f"{name if weighting == 'count' else '':<13}{len(vocab) if weighting == 'count' else '':>7}" f"{weighting:>11}{a:>12.4f}{d:>10.4f}") print(f"baseline {BASELINE:.4f}; seven hand-counted measures + linear model {BY_HAND_SCORE:.4f}") ``` ``` tokenization vocab weighting class mean linear coarse 156 count 0.9040 0.9500 weighted 0.9300 0.9580 careful 109 count 0.8940 0.9620 weighted 0.9200 0.9580 clipped 89 count 0.8700 0.9500 weighted 0.9040 0.9440 baseline 0.4420; seven hand-counted measures + linear model 0.5500 ``` The last line is the second step of the baseline reading of this course. `feature-selection/02` of the Data Preparation and Feature Engineering course had extracted five numbers by hand from a field note and measured that they added 0.0317 on the held-out set; the task there was binary suspicion prediction. When the same approach is applied here to the four-class event kind task, it gives 0.5500 — 0.1080 above the baseline. The lowest number the bag of words gives on the same held-out set is 0.8700, the highest 0.9620. #### A Step's Contribution Is Not a Single Number Twelve setups make it impossible to answer the question "which step adds how many points" with a single subtraction. The contribution of every step depends on the value at which the others stand, and this dependence can be counted: while one step is changed, all the remaining setups are held fixed and the difference is read separately in each of them. ```python # A step's contribution is not a single number: it is measured separately in every setup of the other steps. STEPS = [("tokenization coarse -> careful", [GRID[("careful", g)][k] - GRID[("coarse", g)][k] for g in ("count", "weighted") for k in (0, 1)]), ("tokenization careful -> clipped", [GRID[("clipped", g)][k] - GRID[("careful", g)][k] for g in ("count", "weighted") for k in (0, 1)]), ("weighting count -> weighted", [GRID[(s, "weighted")][k] - GRID[(s, "count")][k] for s in SPLIT for k in (0, 1)]), ("classifier mean -> linear", [GRID[(s, g)][1] - GRID[(s, g)][0] for s in SPLIT for g in ("count", "weighted")]), ("representation by hand -> bag of words", [GRID[(s, g)][k] - BY_HAND_SCORE for s in SPLIT for g in ("count", "weighted") for k in (0, 1)])] print(f"{'step':<38}{'count':>6}{'lowest':>10}{'highest':>11}{'mean':>10}") for label, f in STEPS: print(f"{label:<38}{len(f):>6}{min(f):>+10.4f}{max(f):>+11.4f}" f"{sum(f) / len(f):>+10.4f}") V, _ = build_vocabulary(TRAIN, SPLIT["careful"]) oov = [sum(w not in V for w in SPLIT["careful"](x["text"])) for x in HELD_OUT] total = sum(len(SPLIT["careful"](x["text"])) for x in HELD_OUT) print(f"vocabulary built from training only: {len(V)} words; in the held-out set " f"out-of-vocabulary tokens {sum(oov)}/{total}") best = max(GRID.items(), key=lambda kv: kv[1][1]) print(f"highest setup {best[0][0]} + {best[0][1]} + linear " f"{best[1][1]:.4f}; {best[1][1] - BASELINE:+.4f} above the baseline") ``` ``` step count lowest highest mean tokenization coarse -> careful 4 -0.0100 +0.0120 -0.0020 tokenization careful -> clipped 4 -0.0240 -0.0120 -0.0165 weighting count -> weighted 6 -0.0060 +0.0340 +0.0140 classifier mean -> linear 6 +0.0280 +0.0800 +0.0500 representation by hand -> bag of words 12 +0.3200 +0.4120 +0.3787 vocabulary built from training only: 109 words; in the held-out set out-of-vocabulary tokens 0/3305 highest setup careful + count + linear 0.9620; +0.5200 above the baseline ``` The largest row of the table is at the bottom and it is the real answer of this lesson: almost all of the difference comes from the **representation**. Replacing the seven hand-counted measures with the bag of words adds at least 0.3200 in all twelve setups. Changing the classifier adds 0.0500 on average, weighting 0.0140. The difference between seven numbers and one hundred and nine numbers is seven and a half times the difference between the class mean and the linear model. The weighting row changes sign and this is the second finding of this lesson. When working with the class mean, the inverse document frequency gives a positive contribution in all three tokenizations, the highest being 0.0340; when working with the linear model it drops to the negative side in two setups. The reason lies in the structure of the procedures: the class mean trusts all words equally, so suppressing the high-frequency stop words helps it. The linear model has already learned that suppression through its weights, and a second suppression coming from outside adds no information. The question "how many points does term weighting add" has no single answer; it is a question asked before the classifier was written. Both tokenization rows are negative. The move from coarse splitting to careful brings the vocabulary down from 156 to 109 and costs 0.0020 on average; suffix clipping brings the vocabulary down to 89 and costs 0.0165. Clipping gathers the forms `sayac`, `sayaci`, `sayacin` (meter, the meter, of the meter) into a single token — but it also erases the distinction between "kapak temiz" (cover clean) and "kapakta iz yok" (no mark on the cover), because their distinguishing words intersect at five characters. Shrinking the vocabulary is a measure of cost, not a measure of performance; the two do not have to move in the same direction. The last two lines show the cost of the pipeline rule in this set. Because the vocabulary is built from the training set only, some words of the held-out set would be expected to remain outside; here the number of out-of-vocabulary tokens is zero out of 3,305, because the vocabulary of the fiction is closed and 1,100 notes are enough to see all the phrases. The rule has no cost here, but it was still applied: that the cost is zero is known only after it has been measured. #### Summary - In document-level classification the decision unit is the document; the number of tokens of the input is variable, and the output is a single label. - The domain baseline is the most frequent class and gives 0.4420; seven hand-counted text measures give 0.5500, the bag of words with a linear model gives 0.9620. - Almost all of the difference comes from the representation: the move from hand counting to the bag of words adds at least 0.3200 in all twelve setups, the classifier step adds 0.0500 on average. - A step's contribution is not a single number: term weighting rises to 0.0340 with the class mean and drops to minus 0.0060 with the linear model. - Shrinking the vocabulary does not raise performance: suffix clipping brings the vocabulary from 109 down to 89 and costs 0.0165 on average. #### Next Step In this lesson the bag of words passed the baseline by 0.5200, but 1,100 labeled notes were spent for it. Every note is the counterpart of a field visit, and there can be a hundred of them at hand. Unlabeled notes, on the other hand, are plentiful: the measurement network produces thousands of notes every period and none of them has an event kind written on it. A **proxy task** can be built from these notes — a word is hidden and predicted from its neighbors — and the representation learned there can be carried over to the labeled task. The next lesson measures this loop: how far an embedding pretrained with masked prediction passes a setup trained from scratch as the number of labeled examples shrinks, and at which number of examples the difference closes. ### Pretraining and Fine-Tuning Source: https://academia.sh/en/courses/nlp-and-vision/pretraining-and-fine-tuning Carrying over to the labeled task an embedding table learned by masked prediction from unlabeled notes, and reading the difference it adds together with the number of labels: the proxy task raises its own baseline of 0.0419 to 0.3450 and spends zero labels for it. While the domain baseline is 0.4420, training from scratch with twenty labels gives 0.5850, pretraining and fine-tuning give 0.7480; training from scratch reaches this number only with forty labels. With one hundred sixty labels the difference turns to minus 0.0090. The control setup that does not learn — freezing a random table — stays below 0.51 in all five measurements and its two runs differ by 0.1320; the two runs of the pretrained table differ by 0.0020. In the previous lesson the bag of words passed the domain baseline by 0.5200, but 1,100 labeled notes were spent for it. Every label is the counterpart of the time an inspector spent in the field, and there can be twenty of them at hand. Unlabeled notes, on the other hand, are plentiful: the measurement network produces thousands of notes every period and none of them has an event kind written on it. The question of this lesson is this: can a **proxy task** be built from this unlabeled pile, and when the representation learned there is carried over to the labeled task, **with how many labels** does it add how much of a difference. The proxy task is masked prediction: one word of a note is hidden and predicted from the remaining words. The correct answer is in the text itself, and no label from outside is needed. The attention layer, multi-head attention and positional encoding were built and measured in the Deep Learning course, `architectures/06`–`07`; what is measured here is not the architecture. What is measured is **the loop**: pretraining first, then fine-tuning. - **NL48.** The text fiction is the same as that of the previous lesson and is **fiction**; seed 20260218. - **NL49.** The set is divided into three: the first 700 notes are the **unlabeled pool** (only their text is used, the event kind is never read), the next 400 notes are the **labeled pool**, the last 500 notes are the **held-out set**. The held-out set is the same 500 notes as in the previous lesson. - **NL50.** The vocabulary is built **from the unlabeled pool only**; words that occur in at least two notes are taken. - **NL51.** **The model is written within the lesson**: an embedding table of six numbers per word, the average of the embeddings of the note, and a four-class linear body. The table has 654 parameters, the body 28. - **NL52.** **The baseline of the proxy task is the most frequent word**: the procedure that writes the most frequent word of the vocabulary at every masked position. - **NL53.** Four setups are compared: from scratch (the table is initialized randomly and learned), **random table, frozen** (not learned — the control setup), pretrained and frozen, pretrained and fine-tuned. - **NL54.** The number of labeled examples is measured at five values: 10, 20, 40, 80, 160. The held-out set is the same in all five. - **NL55.** Every number is printed with **two runs**; the runs differ in the initialization seed of the body. #### The Unlabeled Pile and Three Separations The only thing pretraining requires is the text itself. For this reason the separation here is threefold: a pool where the label is never read, a pool where the label is spent by counting, and a held-out set that none of them touches. ```python # FICTIONAL field notes — an extended version of the note field of the Data Preparation and Feature Engineering course. Seed 20260218. import math SEED, M32 = 20260218, 0xFFFFFFFF KINDS = ("ordinary", "unreachable", "meter fault", "illicit use") SHARES = (0.46, 0.22, 0.19, 0.13) PHRASES = {t: s.split("|") for t, s in zip(KINDS, ( "okuma normal|okuma normaldi|sayac erisimi saglandi|sayaca erisildi|olcum tekrarlandi|" "olcumu yineledik|vana kontrol edildi|vanada sorun yok|kapak temiz|kapakta iz yok|" "gosterge okunakli|gostergede rakam net", "abone evde degildi|abone bulunamadi|erisim saglanamadi|erisim yok|kapi zili calismiyor|" "zil calismadi|bahce kapisi kilitli|bahce kapisi kapali|bodrum katina inilemedi|" "bodrum kilitli|adres bulunamadi|adres eksik", "gosterge donmuyor|gostergede hareket yok|sayac cami kirik|sayacin cami catlak|" "rakam okunmuyor|rakamlar silik|ibre sabit kalmis|ibre kilitli|mekanizma takilmis|" "mekanizmada takilma|sayac durmus|sayac calismiyor", "muhur izi var|muhurde iz goruldu|muhur kopmus|muhurler kopuk|baglanti degistirilmis|" "baglantida degisiklik|kelepce gevsek|kelepcede gevseme|boru ek yeri yeni|boruda yeni ek|" "sayac ters takilmis|sayaci ters takmislar"))} NOISE = ("arac park halinde|kapak kirli|hava yagisli|adres teyit edildi|not kisa tutuldu|" "ikinci ziyaret|bahce icinde|kaldirim uzerinde").split("|") REGIONS = "kuzey hatti|guney hatti|dogu hatti|bati hatti|merkez hatti".split("|") SUSPICION_P = {"illicit use": 0.86, "meter fault": 0.21, "unreachable": 0.07, "ordinary": 0.03} def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def build_note(n): r = rng(SEED + 41000 + n) u, k = r(), 0.0 for i, p in enumerate(SHARES): k += p if u < k: t = KINDS[i] break p = [PHRASES[t][int(r() * 12)]] if r() < 0.55: p.append(PHRASES[t][int(r() * 12)]) if r() < 0.34: # side record borrowed from another kind p.append(PHRASES[KINDS[int(r() * 4)]][int(r() * 12)]) if r() < 0.62: p.append(NOISE[int(r() * 8)]) if r() < 0.30: p.append(REGIONS[int(r() * 5)]) return {"text": ", ".join(p), "kind": t, "suspicious": 1 if r() < SUSPICION_P[t] else 0} NOTES = [build_note(i) for i in range(1600)] KIND_INDEX = {t: i for i, t in enumerate(KINDS)} POOL, LABELED, HELD_OUT = NOTES[:700], NOTES[700:1100], NOTES[1100:] def split(m): return [w.strip(",").lower() for w in m.split() if w.strip(",")] d = {} for x in POOL: for w in set(split(x["text"])): d[w] = d.get(w, 0) + 1 S = {w: i for i, w in enumerate(sorted(w for w, c in d.items() if c >= 2))} V, EMB = len(S), 6 SEQUENCES = [z for z in ([S[w] for w in split(x["text"]) if w in S] for x in POOL) if len(z) >= 3] counts = [0] * V for z in SEQUENCES: for i in z: counts[i] += 1 MOST_COMMON_WORD = max(range(V), key=lambda i: counts[i]) kind_counts = {} for x in LABELED[:160]: kind_counts[x["kind"]] = kind_counts.get(x["kind"], 0) + 1 MOST_COMMON = max(kind_counts, key=kind_counts.get) BASELINE = sum(x["kind"] == MOST_COMMON for x in HELD_OUT) / len(HELD_OUT) print(f"unlabeled pool {len(SEQUENCES)} notes, labeled pool {len(LABELED)}, " f"held-out {len(HELD_OUT)}") print(f"vocabulary {V}, embedding size {EMB}, table parameters {V * EMB}, " f"body parameters {4 * EMB + 4}") print(f"domain baseline (most frequent class '{MOST_COMMON}') {BASELINE:.4f}") ``` ``` unlabeled pool 664 notes, labeled pool 400, held-out 500 vocabulary 109, embedding size 6, table parameters 654, body parameters 28 domain baseline (most frequent class 'ordinary') 0.4420 ``` The parameter counts show the tension of this lesson: the table has 654, the body 28. Learning all 682 parameters with ten labeled examples is a situation where the capacity exceeds the data many times over, and the Introduction to Machine Learning course, `generalization/02`, had measured this. The claim of pretraining is this: 654 parameters can be learned from unlabeled text, which leaves 28 parameters to be learned. #### Proxy Task: Masked Prediction In masked prediction every position of every note is hidden in turn, and the hidden word is predicted from the average of the embeddings of the remaining words. What is learned is two tables: the input embedding and the output embedding. Only the first is carried over to the real task. ```python def make_table(seed, scale=0.5): r = rng(seed) return [[(r() + r() + r() - 1.5) * scale for _ in range(EMB)] for _ in range(V)] def train_masked(seed, epochs=4, step=0.1): # THE MODEL: proxy task T, U, c = make_table(seed), make_table(seed + 7, 0.3), [0.0] * V for _ in range(epochs): for z in SEQUENCES: for k in range(len(z)): target, context = z[k], z[:k] + z[k + 1:] h = [sum(T[i][j] for i in context) / len(context) for j in range(EMB)] o = [sum(U[i][j] * h[j] for j in range(EMB)) + c[i] for i in range(V)] e = [math.exp(p - max(o)) for p in o] s = sum(e) dh = [0.0] * EMB for i in range(V): f = (e[i] / s - (1.0 if i == target else 0.0)) * step c[i] -= f for j in range(EMB): dh[j] += f * U[i][j] U[i][j] -= f * h[j] for i in context: for j in range(EMB): T[i][j] -= dh[j] / len(context) return T, U, c def score_masked(T, U, c): hit = base = total = 0 for z in SEQUENCES[:300]: for k in range(len(z)): target, context = z[k], z[:k] + z[k + 1:] h = [sum(T[i][j] for i in context) / len(context) for j in range(EMB)] o = [sum(U[i][j] * h[j] for j in range(EMB)) + c[i] for i in range(V)] hit += max(range(V), key=lambda i: o[i]) == target base += MOST_COMMON_WORD == target total += 1 return hit / total, base / total T0, U0, c0 = train_masked(SEED) mk, mt = score_masked(T0, U0, c0) print(f"masked prediction: baseline (most frequent word) {mt:.4f}, proxy task {mk:.4f}, " f"difference {mk - mt:+.4f}") print(f"labels spent 0; pretraining {4} epochs, {len(SEQUENCES)} notes") ``` ``` masked prediction: baseline (most frequent word) 0.0419, proxy task 0.3450, difference +0.3032 labels spent 0; pretraining 4 epochs, 664 notes ``` On its own measure the proxy task passes its baseline by 0.3032, and the number of labels spent is zero. The Introduction to Machine Learning course, `fundamental-concepts/03`, had built the same structure with a numerical proxy task, and there too the proxy task had passed its own baseline — but when the contribution of the carried-over representation was measured, the share of the learned part had come out **minus** 0.0027. The result of that lesson was this: passing the baseline on the proxy task does not bring passing the baseline on the real task. The same question is asked again here and its answer is measured again. #### The Difference It Adds and With How Many Labels The real task is the task of the previous lesson: the event kind of the note. The only thing that changes is how many labels are at hand. The control setup in the fourth line is the real instrument of this section: if a randomly initialized and **never trained** table is frozen and used with the same body, how much of the number comes from averaging itself. ```python def encode(x): return [S[w] for w in split(x["text"]) if w in S] or [MOST_COMMON_WORD] LABELED_ENC = [(encode(x), KIND_INDEX[x["kind"]]) for x in LABELED] HELD_ENC = [(encode(x), KIND_INDEX[x["kind"]]) for x in HELD_OUT] def train(T, D, seed, learn, epochs=150, step=3.0): # THE MODEL T = [u[:] for u in T] r = rng(seed) W = [[(r() - 0.5) * 0.2 for _ in range(EMB)] for _ in range(4)] b, n = [0.0] * 4, len(D) for _ in range(epochs): gW = [[0.0] * EMB for _ in range(4)] gb = [0.0] * 4 gT = {} if learn else None for z, y in D: h = [sum(T[i][j] for i in z) / len(z) for j in range(EMB)] o = [sum(W[q][j] * h[j] for j in range(EMB)) + b[q] for q in range(4)] e = [math.exp(p - max(o)) for p in o] s = sum(e) dh = [0.0] * EMB for q in range(4): f = e[q] / s - (1.0 if q == y else 0.0) gb[q] += f for j in range(EMB): gW[q][j] += f * h[j] dh[j] += f * W[q][j] if learn: for i in z: g = gT.setdefault(i, [0.0] * EMB) for j in range(EMB): g[j] += dh[j] / len(z) for q in range(4): b[q] -= step * gb[q] / n for j in range(EMB): W[q][j] -= step * gW[q][j] / n if learn: for i, g in gT.items(): for j in range(EMB): T[i][j] -= step * g[j] / n return T, W, b def accuracy(P, D): T, W, b = P return sum(max(range(4), key=lambda q: sum( W[q][j] * sum(T[i][j] for i in z) / len(z) for j in range(EMB)) + b[q]) == y for z, y in D) / len(D) SIZES = (10, 20, 40, 80, 160) SETUPS = (("from scratch", False, True), ("random table, frozen", False, False), ("pretrained, frozen", True, False), ("pretrained, fine-tuned", True, True)) RESULTS = {} print(f"{'setup':<25}{'learned':>9}" + "".join(f"{'n=' + str(n):>9}" for n in SIZES) + f"{'runs':>8}") for label, pretrained, learn in SETUPS: RESULTS[label] = {} for n in SIZES: RESULTS[label][n] = [accuracy(train(T0 if pretrained else make_table(SEED + 900 + k), LABELED_ENC[:n], SEED + 30 + k, learn), HELD_ENC) for k in (1, 2)] print(f"{label:<25}{4 * EMB + 4 + (V * EMB if learn else 0):>9}" + "".join(f"{sum(RESULTS[label][n]) / 2:>9.4f}" for n in SIZES) + f"{max(abs(v[0] - v[1]) for v in RESULTS[label].values()):>8.4f}") print(f"domain baseline {BASELINE:.4f} (last column: the largest difference of the two runs)") for label, n in (("pretrained, frozen", 10), ("pretrained, fine-tuned", 20)): h = sum(RESULTS[label][n]) / 2 y = [m for m in SIZES if sum(RESULTS["from scratch"][m]) / 2 >= h] print(f"{label} n={n} -> {h:.4f}; from scratch reaches the same number with " f"{y[0] if y else '>160'} examples") ``` ``` setup learned n=10 n=20 n=40 n=80 n=160 runs from scratch 682 0.5050 0.5850 0.7770 0.8740 0.9160 0.0460 random table, frozen 28 0.4510 0.4630 0.5000 0.5050 0.5000 0.1320 pretrained, frozen 28 0.6050 0.6610 0.7470 0.7470 0.7550 0.0020 pretrained, fine-tuned 682 0.5880 0.7480 0.8180 0.8780 0.9070 0.0080 domain baseline 0.4420 (last column: the largest difference of the two runs) pretrained, frozen n=10 -> 0.6050; from scratch reaches the same number with 40 examples pretrained, fine-tuned n=20 -> 0.7480; from scratch reaches the same number with 40 examples ``` The second row is the most important row of this table, and it has to be written before reading. When a randomly initialized table that has learned nothing is frozen, it stays below 0.51 in five out of five measurements — about 0.06 above the baseline. Averaging itself, without learning which vector the words sit on, gives almost nothing. The 0.7550 of the third row is therefore the counterpart of **what masked prediction learned**: the same 28 parameters, the same averaging, the only difference being that the table came from unlabeled text, and the difference is 0.2550. This is the reverse of the result of the Introduction to Machine Learning course, `fundamental-concepts/03`. There the learned part of the proxy task had added minus 0.0027 to the real task, and the whole of the contribution had come from the carried-over raw observation. Here there is no carried-over raw observation — what is carried over is only the learned table, and the contribution is 0.2550. The difference between the two lessons lies in **how much** the proxy task is related: the link between predicting the reading of a period and suspected illicit use is weak, while the link between predicting the hidden word and the event kind is direct, because both come from the same word co-occurrences. The label count columns give the second answer. With twenty labels the fine-tuned setup gives 0.7480 and training from scratch 0.5850: the difference is 0.1630, and since it is larger than three times the run variability of 0.0460, it counts as measured. Training from scratch reaches the same number only with forty labels. At one hundred sixty, from scratch gives 0.9160 and fine-tuned 0.9070 — the difference has turned to minus 0.0090, that is, it has closed. What pretraining gained is not the final number but the twenty labels that did not have to be collected to reach it. The last column stands in the table separately. The two runs of the random frozen setup differ by 0.1320: every number printed in that row is an accident that depends on which random table happened to fall. The same body with the pretrained table differs by 0.0020. Pretraining does not only raise the average, it also **lowers the scatter between runs**; this second thing is never visible in a table that prints a single number. #### Summary - The proxy task is masked prediction and it spends no labels: on its own measure it gives 0.3450 where the baseline is 0.0419, a difference of 0.3032. - The domain baseline is 0.4420; with twenty labels training from scratch gives 0.5850, pretraining and fine-tuning give 0.7480, and training from scratch reaches this number only with forty labels. - The difference closes with the number of labels: at one hundred sixty labels the fine-tuned setup falls 0.0090 below training from scratch. - The control setup separates the source of the difference: when a random table that has never been trained is frozen, it stays below 0.51 in all five measurements; the pretrained table gives 0.7550 with the same 28 parameters. - Pretraining also lowers the scatter between runs: the two runs of the random table differ by 0.1320, the two runs of the pretrained table by 0.0020. #### Next Step In six lessons the output of the model was always a **label**: a class number, or a class number per token. The correct answer was at hand and the measure was a single comparison. When the output of a model is **the text itself**, this measure cannot be built: there is no single correct sequence for the summary of a note, and other sequences carrying the same content are also correct. In this case the criterion turns into the **overlap** between the candidate text and the reference text. The next lesson counts what this criterion rewards and shows this: a procedure that gives the input back as it is, one that has learned nothing, gets a high number on an overlap-based criterion. ### Text Generation and Summarization Source: https://academia.sh/en/courses/nlp-and-vision/text-generation-and-summarization Counting what the metric rewards on tasks where the output is text rather than a label: a procedure that gives the input back as it is and has learned nothing gets 1.0000 on the recall side of the overlap-based metric and ranks first among four procedures. The same procedure gets 0.6362 on the precision side and 0.7777 in balanced form, and falls to third place. When eight unrelated words are appended to the end of the copy, recall stays at 1.0000 while precision drops from 0.6362 to 0.2700. The generator procedure, which reports the content of the reference correctly but writes it with other words, gets 0.0684 in balanced form and 0.0114 on 2-grams; on the content check it is second with 0.9320. In six lessons the output of the model was always a label and the correct answer was at hand. When a summary of a field note is requested, this setup collapses: there are several correct summaries that carry the same content, and none of them is written in a list of "correct answers". The summary the inspector wrote is an example, not the single correct one. In this case the metric turns into the **overlap** between the candidate text and the reference text: how many words and how many word pairs are shared. The metric thus becomes computable — but what it rewards is a separate question, and this lesson answers that question with a number. The harshest form of the answer is this: a procedure that gives the input back as it is, one that has learned nothing, gets the **highest number** that can be obtained on this metric. - **NL56.** The text fiction is the same as that of the previous lessons and is **fiction**; seed 20260218. - **NL57.** **The reference summary is also fiction** and is produced by a rule: the phrases that come from the kind of the note itself are kept; the borrowed side record, the noise phrase and the line marker are dropped. This means that the summary is built from the words of the note itself, and it is the source of the result of the lesson. - **NL58.** The fiction assumes that the phrase that reports the decision is written **at the beginning of the note**. The number of the "first phrase" procedure on the content measure is a consequence of this assumption, not a superiority of method. - **NL59.** Overlap is computed separately in two directions: the **precision side** (how many of the candidate's n-grams exist in the reference) and the **recall side** (how many of the reference's n-grams exist in the candidate). The balanced form is the harmonic mean of the two. The metric is measured on single words (1-grams) and word pairs (2-grams). - **NL60.** Repeated n-grams are counted by **clipping**: an n-gram in the candidate is counted as many times as it occurs in the reference. Otherwise a candidate that repeats the same word would inflate precision. - **NL61.** **The domain baseline is to give the input back as it is.** In a summarization task this is the dumbest procedure: it makes no choice, learns nothing, and writes the input as the output. - **NL62.** **The content check is a number separate from the metric**: is the set of event kinds the candidate points to equal to the real kind of the note. This number measures what the overlap metric cannot see. - **NL63.** The weights of the selector and generator procedures are learned **from the training set only**; the held-out set is the last 500 notes. #### Reference Text and Candidate Text The metric has two ends: the **reference text** written by the inspector and the **candidate text** produced by the procedure. The fiction builds the reference text by selecting from the phrases of the note itself — as real summarizers often do. The cost of this selection appears at the end of the lesson. ```python # FICTIONAL field notes — an extended version of the note field of the Data Preparation and Feature Engineering course. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF KINDS = ("ordinary", "unreachable", "meter fault", "illicit use") SHARES = (0.46, 0.22, 0.19, 0.13) PHRASES = {t: s.split("|") for t, s in zip(KINDS, ( "okuma normal|okuma normaldi|sayac erisimi saglandi|sayaca erisildi|olcum tekrarlandi|" "olcumu yineledik|vana kontrol edildi|vanada sorun yok|kapak temiz|kapakta iz yok|" "gosterge okunakli|gostergede rakam net", "abone evde degildi|abone bulunamadi|erisim saglanamadi|erisim yok|kapi zili calismiyor|" "zil calismadi|bahce kapisi kilitli|bahce kapisi kapali|bodrum katina inilemedi|" "bodrum kilitli|adres bulunamadi|adres eksik", "gosterge donmuyor|gostergede hareket yok|sayac cami kirik|sayacin cami catlak|" "rakam okunmuyor|rakamlar silik|ibre sabit kalmis|ibre kilitli|mekanizma takilmis|" "mekanizmada takilma|sayac durmus|sayac calismiyor", "muhur izi var|muhurde iz goruldu|muhur kopmus|muhurler kopuk|baglanti degistirilmis|" "baglantida degisiklik|kelepce gevsek|kelepcede gevseme|boru ek yeri yeni|boruda yeni ek|" "sayac ters takilmis|sayaci ters takmislar"))} NOISE = ("arac park halinde|kapak kirli|hava yagisli|adres teyit edildi|not kisa tutuldu|" "ikinci ziyaret|bahce icinde|kaldirim uzerinde").split("|") REGIONS = "kuzey hatti|guney hatti|dogu hatti|bati hatti|merkez hatti".split("|") PHRASE_KIND = {c: t for t in KINDS for c in PHRASES[t]} def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def build_note(n): r = rng(SEED + 41000 + n) u, k = r(), 0.0 for i, p in enumerate(SHARES): k += p if u < k: t = KINDS[i] break p = [PHRASES[t][int(r() * 12)]] if r() < 0.55: p.append(PHRASES[t][int(r() * 12)]) if r() < 0.34: p.append(PHRASES[KINDS[int(r() * 4)]][int(r() * 12)]) if r() < 0.62: p.append(NOISE[int(r() * 8)]) if r() < 0.30: p.append(REGIONS[int(r() * 5)]) return {"text": ", ".join(p), "kind": t, "phrases": p, "summary": ", ".join(c for c in p if PHRASE_KIND.get(c) == t)} DATA = [build_note(i) for i in range(1600)] TRAIN, HELD_OUT = DATA[:1100], DATA[1100:] for i in (25, 40): print(f"note : {DATA[i]['text']}") print(f"summary: {DATA[i]['summary']} ({DATA[i]['kind']})") print("avg note", round(sum(len(x["text"].split()) for x in HELD_OUT) / len(HELD_OUT), 2), "avg summary", round(sum(len(x["summary"].split()) for x in HELD_OUT) / len(HELD_OUT), 2)) ``` ``` note : bodrum katina inilemedi, abone evde degildi, sayac cami kirik, bahce icinde summary: bodrum katina inilemedi, abone evde degildi (unreachable) note : vanada sorun yok, gostergede rakam net, sayac durmus, kuzey hatti summary: vanada sorun yok, gostergede rakam net (ordinary) avg note 6.61 avg summary 3.99 ``` The average note is 6.61 tokens, the average summary 3.99 tokens. The summary is about three fifths of the length of the note and all of it comes from inside the note. These two sentences determine in advance the first row of the next table. #### Four Procedures, Six Numbers Four procedures are compared. **Copy the input** writes the note as it is. **First phrase** writes only what comes before the first comma. The **learned selector** counts, for every phrase, the rate of "how many times did this phrase enter the summary in the training set" and keeps those above one half. The **generator procedure** copies no phrase: it predicts the event kind from the phrases of the note and writes the canonical sentence of that kind — its output is built from words that do not occur in the note. ```python def tokenize(m): return [w.strip(",").lower() for w in m.split() if w.strip(",")] def ngrams(z, n): return [tuple(z[i:i + n]) for i in range(len(z) - n + 1)] def overlap(candidate, reference, n=1): a, k = ngrams(tokenize(candidate), n), ngrams(tokenize(reference), n) if not a or not k: return 0.0, 0.0 match = 0 remaining = list(k) for g in a: if g in remaining: remaining.remove(g) match += 1 return match / len(a), match / len(k) # --- procedures def p_copy(x): return x["text"] def p_first(x): return x["phrases"][0] TALLY = {} for x in TRAIN: for c in x["phrases"]: a = TALLY.setdefault(c, [0, 0]) a[0] += 1 a[1] += c in x["summary"].split(", ") SELECT_RATE = {c: (a[1] / a[0]) for c, a in TALLY.items()} def p_selector(x): p = [c for c in x["phrases"] if SELECT_RATE.get(c, 0.0) > 0.5] return ", ".join(p) if p else x["phrases"][0] CANONICAL = {"ordinary": "olagan okuma", "unreachable": "erisim yok", "meter fault": "sayac arizasi", "illicit use": "kacak suphesi"} VOTES = {} for x in TRAIN: for c in x["phrases"]: if c in PHRASE_KIND: VOTES.setdefault(c, {}) VOTES[c][x["kind"]] = VOTES[c].get(x["kind"], 0) + 1 def p_generator(x): s = {} for c in x["phrases"]: for t, v in VOTES.get(c, {}).items(): s[t] = s.get(t, 0) + v t = max(s, key=s.get) if s else "ordinary" return CANONICAL[t] + " bildirildi" POINTS_TO = dict(PHRASE_KIND, **{CANONICAL[t]: t for t in KINDS}) def content_ok(candidate, x): present = {POINTS_TO[c] for c in POINTS_TO if c in candidate} return 1 if present == {x["kind"]} else 0 PROCEDURES = (("copy the input", p_copy), ("first phrase", p_first), ("learned selector", p_selector), ("generator procedure", p_generator)) RESULTS = {} print(f"{'procedure':<19}{'length':>8}{'recall':>9}{'precision':>11}" f"{'balanced':>10}{'2-gram balanced':>17}{'content':>9}") for label, f in PROCEDURES: u = r1 = p1 = r2 = p2 = ic = 0.0 for x in HELD_OUT: a = f(x) u += len(tokenize(a)) p, r = overlap(a, x["summary"], 1) p1 += p r1 += r q2, s2 = overlap(a, x["summary"], 2) p2 += q2 r2 += s2 ic += content_ok(a, x) n = len(HELD_OUT) u, r1, p1, r2, p2, ic = u / n, r1 / n, p1 / n, r2 / n, p2 / n, ic / n g1 = 2 * r1 * p1 / (r1 + p1) if r1 + p1 else 0.0 g2 = 2 * r2 * p2 / (r2 + p2) if r2 + p2 else 0.0 RESULTS[label] = (u, r1, p1, g1, g2, ic) print(f"{label:<19}{u:>8.2f}{r1:>9.4f}{p1:>11.4f}{g1:>10.4f}{g2:>17.4f}{ic:>9.4f}") for j, label in ((1, "recall"), (3, "balanced"), (5, "content")): s = sorted(RESULTS, key=lambda a: -RESULTS[a][j]) print(f"{label:<11} ranking: " + " > ".join(s)) ``` ``` procedure length recall precision balanced 2-gram balanced content copy the input 6.61 1.0000 0.6362 0.7777 0.7265 0.7440 first phrase 2.41 0.6901 1.0000 0.8167 0.7576 1.0000 learned selector 4.60 1.0000 0.8948 0.9445 0.9312 0.7440 generator procedure 3.00 0.0622 0.0760 0.0684 0.0114 0.9320 recall ranking: copy the input > learned selector > first phrase > generator procedure balanced ranking: learned selector > first phrase > copy the input > generator procedure content ranking: first phrase > generator procedure > copy the input > learned selector ``` The first row is the debt this lesson pays to the course. The procedure that gives the input back as it is gets **1.0000** on the recall side — the highest number that can be obtained on the scale — and ranks first among the four procedures. This procedure received no training, has no parameters, and makes no choice. The reason for the number lies in the definition of the fiction: because the reference summary is built from the words of the note itself, a candidate that writes the whole note necessarily covers every n-gram of the reference. The recall side measures **coverage**, and the cheapest way of covering is to write everything. The same procedure gets 0.6362 on the precision side and 0.7777 in balanced form, and falls to third place. When the three ranking lines are read side by side, the order of the four procedures comes out in three different ways on three metrics; the answer to the question "which procedure is better" depends on which number is printed. The last row is the second debt. The generator procedure gets 0.9320 on the content check — second among the four procedures — but 0.0684 in the balanced form of the overlap metric and 0.0114 on 2-grams. The metric can hardly see this procedure at all, because it says the right thing **with other words**. An overlap-based metric measures the sharing of words, not the correctness of the report. The second row carries a warning. The "first phrase" procedure gets 1.0000 on the content check, and this number is not a superiority of method: the fiction assumes that the phrase that reports the decision is written at the beginning of the note, so taking the first phrase never misses the kind. The same procedure stays at 0.6901 on recall, because in some of the notes the reference summary consists of two phrases and the second is dropped. When it is not written which clause of the fiction a number comes from, this row can be read as "the best summarizer is the shortest one"; when the fiction is written, it cannot be read that way. The 2-gram column does not repeat the single word column. The balanced form of the copy drops from 0.7777 to 0.7265, that of the selector from 0.9445 to 0.9312, that of the generator procedure from 0.0684 to 0.0114. As the length of the sequence grows, the metric starts to look not only at the choice of words but also at the **order**, and for a candidate written with other words the penalty grows sixfold. The distance between the 0.9445 of the learned selector and its 0.7440 on the content measure is the other face of the same gap: the selector finds the words of the reference summary accurately but also keeps the borrowed side record, that is, the event kind the summary reports turns out wrong in one event out of four. The metric does not see this, because that side record is also inside the note, and if it is not in the reference it only lowers precision by a small amount. #### What the Metric Rewards The claim that the recall side rewards coverage can be tested directly: words that do not occur in the reference summary at all are appended to the end of the copy, and one looks at which of the numbers moves. ```python # What the metric rewards: what happens when unrelated words are appended to the end of the candidate. FILLER = "kayit alindi ekip donusu form imzalandi saat girildi".split() print(f"{'candidate':<26}{'length':>8}{'recall':>9}{'precision':>11}" f"{'balanced':>10}{'content':>9}") for e in (0, 2, 4, 8): u = r = p = ic = 0.0 for x in HELD_OUT: a = x["text"] + ("" if e == 0 else " " + " ".join(FILLER[:e])) u += len(tokenize(a)) q, s = overlap(a, x["summary"], 1) p += q r += s ic += content_ok(a, x) n = len(HELD_OUT) u, r, p, ic = u / n, r / n, p / n, ic / n print(f"{'copy the input + ' + str(e) + ' filler':<26}{u:>8.2f}{r:>9.4f}" f"{p:>11.4f}{2 * r * p / (r + p):>10.4f}{ic:>9.4f}") r0, p0, g0, i0 = RESULTS["generator procedure"][1], RESULTS["generator procedure"][2], \ RESULTS["generator procedure"][3], RESULTS["generator procedure"][5] print(f"{'generator procedure':<26}{RESULTS['generator procedure'][0]:>8.2f}{r0:>9.4f}" f"{p0:>11.4f}{g0:>10.4f}{i0:>9.4f}") below = [c for c in HELD_OUT if overlap(p_copy(c), c["summary"], 1)[1] < 1.0] print(f"notes where the recall of the copy stays below 1.0000: {len(below)}/{len(HELD_OUT)}") ``` ``` candidate length recall precision balanced content copy the input + 0 filler 6.61 1.0000 0.6362 0.7777 0.7440 copy the input + 2 filler 8.61 1.0000 0.4685 0.6381 0.7440 copy the input + 4 filler 10.61 1.0000 0.3750 0.5454 0.7440 copy the input + 8 filler 14.61 1.0000 0.2700 0.4252 0.7440 generator procedure 3.00 0.0622 0.0760 0.0684 0.9320 notes where the recall of the copy stays below 1.0000: 0/500 ``` The recall column is exactly 1.0000 in all four rows. While the length of the candidate text goes from 6.61 to 14.61, more than doubling, this number does not move by a single digit; the content check does not move either, because the appended words point to no event kind. The only number that moves is precision: it drops from 0.6362 to 0.2700 and pulls the balanced form from 0.7777 to 0.4252. In a report that prints only the recall side, appending unrelated words to the candidate text is a **free** operation. The last row says that the copy got its 1.0000 on five hundred of five hundred notes. This is not an average but an identity: as long as the reference summary of the fiction is a subset of the note, the copy can never fall below 1.0000 on recall. Most real summaries also carry this property, because summarizers reuse the words of the source. In this setup there is no link between a metric rewarding a procedure and that procedure having learned something. #### Summary - When the output is text there is no single correct answer; the metric turns into the overlap between the candidate text and the reference text and is read separately in two directions. - The procedure that gives the input back as it is gets 1.0000 on the recall side and is first among the four procedures; it gets 0.6362 on the precision side and falls to third place. - When eight unrelated words are appended to the end of the copy, recall stays at 1.0000 and precision drops to 0.2700: the recall side of the metric rewards coverage, not correctness. - The metric sees the sharing of words, not the correctness of the report: the generator procedure gets 0.9320 on the content check, while it gets 0.0684 in balanced form and 0.0114 on 2-grams. - The same four outputs give three different rankings on three metrics; which procedure is better depends on which number is printed. #### Next Step In this lesson three metrics gave three different rankings and next to each of them was written what was being rewarded. The same problem exists in all the tasks of this course: a number read at the token level and a number read at the entity level rank the same output in two different ways, and on an imbalanced set of classes overall accuracy and the per-class average rank differently. The next lesson is the close of the topic: **the same four outputs give four different rankings on four metrics**, and which baseline each metric should be read against is written out one by one. ### Natural Language Processing Metrics Source: https://academia.sh/en/courses/nlp-and-vision/natural-language-processing-metrics Reading a single output at four metrics and the four metrics giving four different rankings: the same four systems rank B > A > C > D at the token level, A > B > C > D at the entity level, C > B > A > D at the document level and D > B > A > C on the recall side of overlap, and the first of each metric is a different system. Each metric is read against its own baseline: the most frequent label at the token level 0.3970, the most frequent class at the document level 0.4420, marking no span at all at the entity level 0.0000, marking the whole input on the recall side of overlap 1.0000. In the last row the difference between the baseline and the best system is exactly 0.0000. In the previous lesson three metrics gave three different rankings, and the procedure that copies the input got 1.0000 on the recall side. That finding is not a property of a single task. In the eight lessons of this topic, metrics were read at four different levels: at the token level, at the entity level, at the document level, and through overlap. The same system gives four different numbers at these four levels, and the numbers do not preserve each other's ranking. This lesson is the close of the topic and finishes with a single setup: four systems, **a single output format**, four metrics. The output is one label per token for each note; all four of the metrics are read from the same labeling. The definitions of precision, recall and the balanced form were set up in `evaluation-and-tuning` of the Supervised Learning course and are not repeated here; the question here is not the definition but how the **decision unit** and the **baseline** change the metric. - **NL64.** The text fiction is the same as that of the previous lessons and is **fiction**; seed 20260218. - **NL65.** **The gold labeling is fiction**: all tokens of the phrases that come from the kind of the note itself are labeled with that kind, and all remaining tokens are labeled `none`. The borrowed side record counts as `none` — the event the note reports is not that one. - **NL66.** **The output format is single**: each system produces one label for each token. All four metrics are computed from this single output; no system gives a separate output for a metric. - **NL67.** A **span** is the longest run of consecutive tokens with the same label. At the entity level a span counts as correct only if its start, its end and its label all match. - **NL68.** **The document-level decision is derived from the output**: the most frequent label among the marked tokens counts as the kind of the note; if there is no mark at all, the most frequent class is written. - **NL69.** The overlap metric is the **recall side** of the previous lesson: how many of the tokens that are marked in the gold labeling are also marked in the output. Repeated tokens are counted by clipping. - **NL70.** **The four metrics have four different baselines**, and each baseline is the dumbest procedure of that metric: writing the most frequent label to every token, marking no span at all, writing the most frequent class, marking the whole input. - **NL71.** The tables the systems learn are built **from the training set only**; the held-out set is the last 500 notes. - **NL72.** None of the four systems is derived from another; all four run on the same held-out set, with the same output format. #### One Output, Four Readings Once the gold labeling is set up, two baselines can be read directly: the procedure that writes the most frequent label to every token at the token level, and the procedure that writes the most frequent class at the document level. ```python # FICTIONAL field notes — an extended version of the note field of the Data Preparation and Feature Engineering course. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF KINDS = ("ordinary", "unreachable", "meter fault", "illicit use") SHARES = (0.46, 0.22, 0.19, 0.13) PHRASES = {t: s.split("|") for t, s in zip(KINDS, ( "okuma normal|okuma normaldi|sayac erisimi saglandi|sayaca erisildi|olcum tekrarlandi|" "olcumu yineledik|vana kontrol edildi|vanada sorun yok|kapak temiz|kapakta iz yok|" "gosterge okunakli|gostergede rakam net", "abone evde degildi|abone bulunamadi|erisim saglanamadi|erisim yok|kapi zili calismiyor|" "zil calismadi|bahce kapisi kilitli|bahce kapisi kapali|bodrum katina inilemedi|" "bodrum kilitli|adres bulunamadi|adres eksik", "gosterge donmuyor|gostergede hareket yok|sayac cami kirik|sayacin cami catlak|" "rakam okunmuyor|rakamlar silik|ibre sabit kalmis|ibre kilitli|mekanizma takilmis|" "mekanizmada takilma|sayac durmus|sayac calismiyor", "muhur izi var|muhurde iz goruldu|muhur kopmus|muhurler kopuk|baglanti degistirilmis|" "baglantida degisiklik|kelepce gevsek|kelepcede gevseme|boru ek yeri yeni|boruda yeni ek|" "sayac ters takilmis|sayaci ters takmislar"))} NOISE = ("arac park halinde|kapak kirli|hava yagisli|adres teyit edildi|not kisa tutuldu|" "ikinci ziyaret|bahce icinde|kaldirim uzerinde").split("|") REGIONS = "kuzey hatti|guney hatti|dogu hatti|bati hatti|merkez hatti".split("|") PHRASE_KIND = {c: t for t in KINDS for c in PHRASES[t]} def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def build_note(n): r = rng(SEED + 41000 + n) u, k = r(), 0.0 for i, p in enumerate(SHARES): k += p if u < k: t = KINDS[i] break p = [PHRASES[t][int(r() * 12)]] if r() < 0.55: p.append(PHRASES[t][int(r() * 12)]) if r() < 0.34: p.append(PHRASES[KINDS[int(r() * 4)]][int(r() * 12)]) if r() < 0.62: p.append(NOISE[int(r() * 8)]) if r() < 0.30: p.append(REGIONS[int(r() * 5)]) return {"text": ", ".join(p), "kind": t, "phrases": p, "summary": ", ".join(c for c in p if PHRASE_KIND.get(c) == t)} NONE = "none" def tokenize(m): return [w.strip(",").lower() for w in m.split() if w.strip(",")] DATA = [] for i in range(1600): x = build_note(i) z, e = [], [] for c in x["phrases"]: # GOLD label: phrases of its own kind w = tokenize(c) z += w e += [x["kind"] if PHRASE_KIND.get(c) == x["kind"] else NONE] * len(w) x["tokens"], x["gold"] = z, e DATA.append(x) TRAIN, HELD_OUT = DATA[:1100], DATA[1100:] label_counts = {} for x in TRAIN: for u in x["gold"]: label_counts[u] = label_counts.get(u, 0) + 1 MOST_COMMON_LABEL = max(label_counts, key=label_counts.get) MOST_COMMON_KIND = max(KINDS, key=lambda t: sum(y["kind"] == t for y in TRAIN)) n = len(HELD_OUT) BASE_TOKEN = (sum(u == MOST_COMMON_LABEL for x in HELD_OUT for u in x["gold"]) / sum(len(x["gold"]) for x in HELD_OUT)) BASE_DOC = sum(x["kind"] == MOST_COMMON_KIND for x in HELD_OUT) / n print(f"held out {n} notes, {sum(len(x['tokens']) for x in HELD_OUT)} tokens, " f"marked tokens {sum(u != NONE for x in HELD_OUT for u in x['gold'])}") total_labels = sum(label_counts.values()) print("gold label distribution (train): " + " ".join( f"{u}={label_counts[u] / total_labels:.3f}" for u in sorted(label_counts, key=lambda u: -label_counts[u])[:3])) print(" " + " ".join( f"{u}={label_counts[u] / total_labels:.3f}" for u in sorted(label_counts, key=lambda u: -label_counts[u])[3:])) print(f"token-level baseline ('{MOST_COMMON_LABEL}' on every token) {BASE_TOKEN:.4f}, " f"document-level baseline ('{MOST_COMMON_KIND}') {BASE_DOC:.4f}") ``` ``` held out 500 notes, 3305 tokens, marked tokens 1993 gold label distribution (train): none=0.413 ordinary=0.267 unreachable=0.125 meter fault=0.111 illicit use=0.084 token-level baseline ('none' on every token) 0.3970, document-level baseline ('ordinary') 0.4420 ``` The label distribution says why the token-level baseline is 0.3970: about two fifths of the tokens belong to no event, and `none` alone is the most frequent label. The Supervised Learning course had measured how misleading overall accuracy is on an imbalanced class set; here the same imbalance appears the moment the **decision unit** of the metric is chosen. Of the 3,305 tokens, 1,993 are marked and 1,312 are not. #### Four Systems, Four Rankings The four systems represent four different design decisions. **A — phrase-level selector** makes the decision over the phrase: if the rate at which a phrase entered the gold labeling in the training set is above 0.75, all tokens of the phrase are marked, and the label comes from the document prediction. **B — token-level selector** makes the decision over single tokens: if the marking rate of a word is above 0.40, it is marked. **C — narrow marker** marks only the first phrase and takes the label from the kind of that phrase itself. **D — mark everything** writes the most frequent kind to every token; it learns nothing. ```python PHRASE_COUNTS, TOKEN_COUNTS, KIND_SCORES = {}, {}, {} for x in TRAIN: for c in x["phrases"]: a = PHRASE_COUNTS.setdefault(c, [0, 0]) a[0] += 1 a[1] += PHRASE_KIND.get(c) == x["kind"] if c in PHRASE_KIND: KIND_SCORES.setdefault(c, {}) KIND_SCORES[c][x["kind"]] = KIND_SCORES[c].get(x["kind"], 0) + 1 for w, u in zip(x["tokens"], x["gold"]): a = TOKEN_COUNTS.setdefault(w, [0, 0]) a[0] += 1 a[1] += u != NONE PHRASE_RATE = {c: a[1] / a[0] for c, a in PHRASE_COUNTS.items()} TOKEN_RATE = {w: a[1] / a[0] for w, a in TOKEN_COUNTS.items()} def predict_kind(x): s = {} for c in x["phrases"]: for t, v in KIND_SCORES.get(c, {}).items(): s[t] = s.get(t, 0) + v return max(s, key=s.get) if s else MOST_COMMON_KIND def system_phrase(x): t, e = predict_kind(x), [] for c in x["phrases"]: e += [(t if PHRASE_RATE.get(c, 0.0) > 0.75 else NONE)] * len(tokenize(c)) return e def system_token(x): t = predict_kind(x) return [(t if TOKEN_RATE.get(w, 0.0) > 0.40 else NONE) for w in x["tokens"]] def system_narrow(x): e = [] for i, c in enumerate(x["phrases"]): e += [(PHRASE_KIND.get(c, NONE) if i == 0 else NONE)] * len(tokenize(c)) return e def system_all(x): return [MOST_COMMON_KIND] * len(x["tokens"]) def spans(e): a, i = [], 0 while i < len(e): if e[i] == NONE: i += 1 continue j = i while j + 1 < len(e) and e[j + 1] == e[i]: j += 1 a.append((i, j, e[i])) i = j + 1 return a def measure(f): token_hit = token_total = span_hit = span_pred = span_gold = doc_hit = 0 overlap_sum = 0.0 for x in HELD_OUT: e = f(x) token_hit += sum(u == v for u, v in zip(e, x["gold"])) token_total += len(e) g, p = spans(x["gold"]), spans(e) span_hit += sum(u in g for u in p) span_pred += len(p) span_gold += len(g) s = {} for u in e: if u != NONE: s[u] = s.get(u, 0) + 1 doc_hit += (max(s, key=s.get) if s else MOST_COMMON_KIND) == x["kind"] left = [w for w, u in zip(x["tokens"], x["gold"]) if u != NONE] hit = 0 for w in [w for w, u in zip(x["tokens"], e) if u != NONE]: if w in left: left.remove(w) hit += 1 overlap_sum += hit / max(1, sum(u != NONE for u in x["gold"])) pr, rc = span_hit / max(1, span_pred), span_hit / max(1, span_gold) return (token_hit / token_total, 2 * pr * rc / (pr + rc) if pr + rc else 0.0, doc_hit / len(HELD_OUT), overlap_sum / len(HELD_OUT)) SYSTEMS = (("A", "phrase-level selector", system_phrase), ("B", "token-level selector", system_token), ("C", "narrow marker", system_narrow), ("D", "mark everything", system_all)) SCORES = {k: measure(f) for k, _, f in SYSTEMS} METRICS = ("token", "entity", "document", "overlap") print(f"{'':<3}{'system':<24}" + "".join(f"{o:>11}" for o in METRICS)) for k, name, _ in SYSTEMS: print(f"{k:<3}{name:<24}" + "".join(f"{v:>11.4f}" for v in SCORES[k])) for j, o in enumerate(METRICS): print(f"{o:<9} ranking: " + " > ".join( sorted(SCORES, key=lambda a: -SCORES[a][j]))) ``` ``` system token entity document overlap A phrase-level selector 0.8136 0.6779 0.8320 0.8423 B token-level selector 0.8378 0.6036 0.9320 0.9693 C narrow marker 0.7619 0.4000 1.0000 0.6901 D mark everything 0.2732 0.0920 0.4420 1.0000 token ranking: B > A > C > D entity ranking: A > B > C > D document ranking: C > B > A > D overlap ranking: D > B > A > C ``` The four ranking lines are the answer of this lesson. All four metrics read the same four outputs, and in all four the ranking is different: B > A > C > D, A > B > C > D, C > B > A > D, D > B > A > C. The **first of each of the four metrics is also a different system**. The question "which system is better" has no answer in this setup; the question that has an answer is "on which metric". Where the differences come from can be read one by one. The difference between A and B lies in the decision unit: because B decides per token, it leads at the token level with 0.8378, but because it makes its decision independently of its neighbor, it skips a word in the middle of a phrase and splits the span in two; at the entity level it falls to 0.6036, 0.0743 below A. The same output, the same number of correct tokens, two different rankings. C is the system that marks the fewest tokens among all systems, and it gets 1.0000 at the document level. This number is a consequence of the fiction — the phrase that reports the decision is written at the beginning of the note — and cannot be read as a superiority of method. The same system stays at 0.4000 at the entity level, because in a part of the notes the gold span covers two phrases and C never marks the second. The D row is the third payment, in this topic, of the second claim of the course. D learns nothing: it writes the most frequent kind of the training set to every token. At the token level it falls even below the baseline with 0.2732, at the entity level it gets 0.0920 — and on the recall side of the overlap metric it becomes the first of the four systems with **1.0000**. If a metric can put a procedure that has learned nothing in first place, printing that metric alone is not a measurement but a choice. #### Each Metric's Own Baseline The four metrics have four different baselines, and the baselines are not on the same scale. The size of a number says nothing by itself; what says something is how far that number stands from its own baseline. ```python BASELINES = (("token-level accuracy", "write the most frequent label to every token", BASE_TOKEN), ("entity-level balanced form", "mark no span", 0.0), ("document-level accuracy", "write the most frequent class", BASE_DOC), ("overlap recall side", "mark the whole input", 1.0)) print(f"{'metric':<28}{'baseline procedure':<46}{'baseline':>9}{'best':>8}{'difference':>11}") for j, (o, procedure, base) in enumerate(BASELINES): best = max(SCORES, key=lambda a: SCORES[a][j]) print(f"{o:<28}{procedure:<46}{base:>9.4f}{SCORES[best][j]:>8.4f}" f"{SCORES[best][j] - base:>+11.4f}") ``` ``` metric baseline procedure baseline best difference token-level accuracy write the most frequent label to every token 0.3970 0.8378 +0.4408 entity-level balanced form mark no span 0.0000 0.6779 +0.6779 document-level accuracy write the most frequent class 0.4420 1.0000 +0.5580 overlap recall side mark the whole input 1.0000 1.0000 +0.0000 ``` The table gives four different distances in four rows. At the entity level the best system is 0.6779 above the baseline, at the document level 0.5580, at the token level 0.4408. These three rows can be ranked but cannot be compared: three different decision units, three different baselines, three different scales. The last row is the closing sentence of the topic. On the recall side of the overlap metric the baseline is 1.0000, because the procedure that marks the whole input necessarily covers every token of the gold labeling. The best system also gets 1.0000, and the difference is exactly **0.0000**. This metric has a baseline that nothing can be added on top of: no learning can pass it, only tie it. The first question to ask when choosing a metric is what the dumbest procedure of that metric gets on the same metric; the second is whether any room is left above that number to pass. The baseline of 0.0000 at the entity level, on the other hand, is at the opposite end and it also misleads: a procedure that marks no span gets zero, so passing the baseline on this metric is free. The more honest baseline is system D in the same table: the procedure that counts the whole note as a single span gets 0.0920, and the difference of the best system drops from 0.6779 to 0.5859. #### Summary - A single token labeling can be read at four different levels, and the four metrics produce four different numbers from the same output. - The four metrics give four different rankings, and the first of the four is four different systems: B at the token level, A at the entity level, C at the document level, D on overlap. - The decision unit determines the metric: the system that decides per token leads at the token level and falls 0.0743 behind at the entity level because it splits the span. - The system that learns nothing falls below the baseline at the token level but becomes first with 1.0000 on the recall side of overlap. - Each metric has its own baseline, and the baselines are not on the same scale: 0.3970, 0.0000, 0.4420 and 1.0000; on the last one the difference between the baseline and the best system is 0.0000. #### Next Step In eight lessons the input was always a **sequence**. The tokens were ordered from left to right, the neighbor of a word lay on a single axis, a span was defined by a start and an end index, and the order itself carried information: "muhur izi var" (seal mark present) and "var izi muhur" (present mark seal) have the same bag of words but are not the same thing. That the overlap metric differs on 2-grams from single words was also a result of this single axis. In a domain where the input is **two-dimensional** and the neighborhood opens not to the right and left but in **every direction**, every part of this setup is asked again. A span is no longer a start and an end but a rectangle; overlap is no longer a counter but a ratio of areas; there is no such thing as a vocabulary, because the unit of the input is not a word but a number, and that number has no meaning on its own. The next topic begins with images and opens with the same question: what is the dumbest procedure of this domain, and what does that procedure get on its own metric. ## Computer Vision Measuring each task against its own baseline on fictional images generated from the coverage grid: the contribution of normalization and augmentation and the transform that breaks the label in augmentation, end-to-end training against the mean image baseline and how many examples it takes for transfer to add something, the number a fixed box placed in the middle of the image gets on the overlap metric and the fact that both the method and the fixed box drop as the threshold rises from 0.3 to 0.7, the baseline that assigns every pixel to the dominant class scoring high because of pixel imbalance and semantic segmentation being unable to separate two adjacent objects, the inconsistency of independent per-frame decisions against a decision that uses time across consecutive frames, the success of a joint representation that matches a note with its image against random matching, and the share a single modality gives on its own. ### Image Data and Preprocessing Source: https://academia.sh/en/courses/nlp-and-vision/image-data-and-preprocessing Measuring how much the way a pixel is turned into a number carries: when the same two procedures are read on three representations, dataset-wide scaling moves no number (0.5500 and 0.6937 stay as they are), while per-image normalization raises the threshold rule from 0.6937 to 0.9250 and lowers the mean image baseline from 0.5500 to 0.4875. Of five augmentation transforms, the quarter turn flips the label in 160 of 160 examples, and the shift breaks it in 1. With forty labels, nearest neighbor reads 0.5938 and 0.6250 without augmentation, two flips read 0.6875 and 0.6250, and a quarter turn added with its label kept reads 0.5188 and 0.5000: below the 0.5312 of the most frequent class. In eight lessons the input was a **sequence**. Tokens ran from left to right, a token's neighbors were the ones before and after it, and the order itself carried information. An image has no single order: a pixel's neighbors are to its right, to its left, above, below, and at its four corners, and none of them comes before the others. What is sought is not in a single cell either but in a pattern formed by adjacent cells, and that pattern can be found anywhere in the grid. The second difference is quieter. In text a token is a **symbol**; the same token in two separate documents is exactly the same thing. A pixel, on the other hand, is a **measurement**: the same object is written with different numbers when the measuring conditions change. So in images **normalization** means transforming pixel values, and it has nothing to do with the text normalization of the Data Collection and Preparation course, which brings a string into canonical form. This is the measure of the lesson — how far the decision on how to turn a pixel into a number moves the same procedure's number, with no model changed. - **CV1. The image is fiction.** The 12x12 coverage grid of the Visualization, Reporting and Analysis at Scale course stands in here for a small single-channel, gray-level image. Each cell is a **pixel**, and its value is a **gray level** between 0 and 255. Seed 20260218. - **CV2. Background and contrast vary from image to image;** this is a difficulty the fiction puts in on purpose. - **CV3. The object is fiction.** A single rectangular **block** is planted in the grid; its sides are between 2 and 6 and differ from each other. - **CV4. The label is the block's orientation.** 1 if the width is greater than the height, 0 otherwise. The label depends not on **where** the block is but on its **shape**. - **CV5. The block gathers toward the middle of the grid.** The position is chosen by the mean of three independent draws; this is the counterpart of the center bias in real image sets, and it will have a cost in the third lesson. - **CV6. The split is 160/80/160 over 400 images** and is fixed throughout the topic. - **CV7. The domain baseline is two procedures:** stating the most frequent class, and the **mean image** — class means are computed, and an image is written to whichever one it is closer to. - **CV8. Threshold and tuning decisions are read from the validation set;** the sweep does not exceed nine candidates. No library is called; the procedures are written inside the lesson. #### Image, Pixel, and Channel ```python # image.py — MODEL: Image fiction: the 12x12 coverage grid of the Visualization, # Reporting and Analysis at Scale course stands in here for a small single-channel, # gray-level image. Each cell is a pixel and its value is a gray level between 0 and 255. import statistics SEED, M32, SIZE = 20260218, 0xFFFFFFFF, 12 MIN_SIDE, MAX_SIDE = 2, 6 RAMP = " .:-=+*#%@" def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def image(no): """One coverage image: background, a single rectangular block, and noise.""" r = rng(SEED + 700000 + no) background = 30 + int(r() * 70) contrast = 20 + int(r() * 36) height = MIN_SIDE + int(r() * (MAX_SIDE - MIN_SIDE + 1)) width = MIN_SIDE + int(r() * (MAX_SIDE - MIN_SIDE + 1)) while width == height: width = MIN_SIDE + int(r() * (MAX_SIDE - MIN_SIDE + 1)) i = int((r() + r() + r()) / 3 * (SIZE - height + 1)) # position gathered toward the center j = int((r() + r() + r()) / 3 * (SIZE - width + 1)) g = [[0] * SIZE for _ in range(SIZE)] for a in range(SIZE): for b in range(SIZE): v = background + (contrast if i <= a < i + height and j <= b < j + width else 0) v += int(r() * 61) - 30 g[a][b] = 0 if v < 0 else (255 if v > 255 else v) return g, (i, j, height, width), (1 if width > height else 0) def shuffle(n, seed): r, s = rng(seed), list(range(n)) for i in range(n - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return s def draw(g): for row in g: print(" " + "".join(RAMP[v * len(RAMP) // 256] for v in row)) RAW = [image(i) for i in range(400)] DATA = [RAW[i] for i in shuffle(400, SEED + 40000)] TRAIN, VAL, TEST = DATA[:160], DATA[160:240], DATA[240:] means = [statistics.mean(v for s in g for v in s) for g, _, _ in DATA] g0, k0, e0 = TRAIN[0] print(f"sample image: block row {k0[0]}, column {k0[1]}, {k0[2]}x{k0[3]}, label {e0}") draw(g0) print(f"{SIZE}x{SIZE} pixels = {SIZE * SIZE}, single channel; the same image with three channels " f"would hold {SIZE * SIZE * 3} numbers") print(f"image means between {min(means):.1f} and {max(means):.1f}") print(f"split {len(TRAIN)}/{len(VAL)}/{len(TEST)}") MAJORITY = max(sum(e for _, _, e in TEST), len(TEST) - sum(e for _, _, e in TEST)) / len(TEST) print(f"domain baseline, most frequent class: {MAJORITY:.4f}") ``` ``` sample image: block row 4, column 2, 4x6, label 1 :-:..::::::. ....:.::.::: :.::-.::.-:- ...-.:.::.:. ..=+==++:.-. --=---=--..: :.====---.-: :.=-==+=:..: .-::.:.-.:-: ::.:-:....:: ::..:-::-::: .:-.:--:.-.: 12x12 pixels = 144, single channel; the same image with three channels would hold 432 numbers image means between 32.9 and 108.0 split 160/80/160 domain baseline, most frequent class: 0.5312 ``` In the sample image the block starts at the fourth row, is four rows tall and six columns wide. It is hard to make out in the drawing: because the ramp is spread over the 0-255 range, both the background and the block fall on the same two or three glyphs. The number says so too — the image means wander between **32.9 and 108.0**, so in one image the inside of the block is as bright as the background itself in another. An image of 12x12 pixels and one channel is **144** numbers; with three channels it would hold **432**. A channel is a quantity measured separately for each pixel, and it multiplies the count by its own number. Stating the most frequent class on the test set gives **0.5312**: this is the domain's cheapest baseline. #### Normalization ```python # preprocessing.py — three representations and two procedures. The procedures stay # the same; only the way a pixel is turned into a number changes. def raw_t(g): return g def scale_t(g): # dataset-wide scaling return [[v / 255 for v in s] for s in g] def norm_t(g): # per-image normalization x = [v for s in g for v in s] m = sum(x) / len(x) sd = (sum((v - m) ** 2 for v in x) / len(x)) ** 0.5 return [[(v - m) / sd for v in s] for s in g] def bright_box(x, threshold): """Bounding box of the pixels above the threshold.""" lit = [(a, b) for a in range(SIZE) for b in range(SIZE) if x[a][b] >= threshold] if not lit: return None return (min(a for a, _ in lit), min(b for _, b in lit), max(a for a, _ in lit) - min(a for a, _ in lit) + 1, max(b for _, b in lit) - min(b for _, b in lit) + 1) def threshold_rule(x, threshold): k = bright_box(x, threshold) return 0 if k is None else (1 if k[3] > k[2] else 0) def mean_image(train, transform): """Domain baseline: class means, the decision is the nearest mean.""" total = {0: [0.0] * (SIZE * SIZE), 1: [0.0] * (SIZE * SIZE)} count = {0: 0, 1: 0} for g, _, e in train: x = [v for s in transform(g) for v in s] for i in range(SIZE * SIZE): total[e][i] += x[i] count[e] += 1 M = {e: [v / count[e] for v in total[e]] for e in (0, 1)} def decide(g): x = [v for s in transform(g) for v in s] d0 = sum((x[i] - M[0][i]) ** 2 for i in range(SIZE * SIZE)) d1 = sum((x[i] - M[1][i]) ** 2 for i in range(SIZE * SIZE)) return 1 if d1 < d0 else 0 return decide def measure(data, decide): return sum(1 for g, _, e in data if decide(g) == e) / len(data) print("same image, normalized per image; same ramp:") draw([[max(0, min(255, int((v + 2) * 51))) for v in s] for s in norm_t(g0)]) REPRESENTATIONS = (("raw pixel", raw_t, [100 + 10 * k for k in range(9)]), ("dataset-wide scaling", scale_t, [(100 + 10 * k) / 255 for k in range(9)]), ("per-image normal.", norm_t, [0.6 + 0.15 * k for k in range(9)])) print(f"\n{'representation':<24}{'threshold':>10}{'mean image':>12}{'threshold rule':>16}") for name, transform, candidates in REPRESENTATIONS: best = max(candidates, key=lambda t: measure(VAL, lambda g: threshold_rule(transform(g), t))) print(f"{name:<24}{best:>10.2f}{measure(TEST, mean_image(TRAIN, transform)):>12.4f}" f"{measure(TEST, lambda g: threshold_rule(transform(g), best)):>16.4f}") ``` ``` same image, normalized per image; same ramp: ===..--:-:-. ..:.-::=:-== -.===:==:+-+ ...=::.-=.=. :.#%##@%:.+: =+%+++%*+:.= -:%*%%+++.+: =.**%*%%=..: .=-:.-:+.=+= -=:-==:..::- ==..=+-=+-== :-+.==+=.+.= representation threshold mean image threshold rule raw pixel 130.00 0.5500 0.6937 dataset-wide scaling 0.51 0.5500 0.6937 per-image normal. 1.65 0.4875 0.9250 ``` The drawing says enough on its own: when each pixel has its own image's mean subtracted and is divided by its own standard deviation, the block becomes visible. The grid, the block, and the noise are the same; only the scale of the numbers changes. The second row of the table gives a **zero** result. Dividing all pixels by 255 leaves both numbers of both procedures as they are: 0.5500 and 0.6937. Applying the **same** factor to every image changes neither the nearest mean nor the threshold ranking; this step is a change of units. Per-image normalization carries the threshold rule **from 0.6937 to 0.9250**: a difference of **0.2313**. A fixed threshold finds the block in an image whose background is 33, and in an image whose background is 108 it folds half of the background into the block too; after normalization, "1.65 standard deviations above the mean" means the same thing in every image. The model did not change, the data did not change, only the **representation** changed. The same representation lowers the mean image baseline **from 0.5500 to 0.4875**. This is not a contradiction: the mean image procedure stays near the baseline in both representations, because the block is in a different place in every image and the orientation is erased when averaging. The 0.5500 on the raw representation is only **0.0188** above the 0.5312 of the most frequent class. The domain baseline draws a lower bound here, not a target. #### Augmentation and the Transform That Breaks the Label ```python # augmentation.py — Augmentation can be measured only on a procedure that learns; the # threshold rule does not learn. The measuring tool is nearest neighbor and here it is # only a tool. def shift(g, di, dj): fill = statistics.median(v for s in g for v in s) return [[g[a - di][b - dj] if 0 <= a - di < SIZE and 0 <= b - dj < SIZE else fill for b in range(SIZE)] for a in range(SIZE)] TRANSFORMS = (("shift (1,1)", lambda g: shift(g, 1, 1)), ("horizontal flip", lambda g: [s[::-1] for s in g]), ("vertical flip", lambda g: g[::-1]), ("quarter turn", lambda g: [[g[SIZE - 1 - b][a] for b in range(SIZE)] for a in range(SIZE)]), ("brightness +25", lambda g: [[min(255, v + 25) for v in s] for s in g])) def box_transform(k, name): i, j, h, w = k if name.startswith("shift"): i, j = i + 1, j + 1 h, w = min(h, SIZE - i), min(w, SIZE - j) elif name == "horizontal flip": j = SIZE - j - w elif name == "vertical flip": i = SIZE - i - h elif name == "quarter turn": i, j, h, w = j, SIZE - i - h, w, h return (i, j, h, w) print(f"{'transform':<17}{'label broken':>16}") for name, _ in TRANSFORMS: broken = sum(1 for _, k, e in TRAIN if (1 if box_transform(k, name)[3] > box_transform(k, name)[2] else 0) != e) print(f"{name:<17}{f'{broken}/{len(TRAIN)}':>16}") def features(g): # normalize, 2x2 mean pool x = [v for s in norm_t(g) for v in s] return [sum(x[(2 * a + p) * SIZE + 2 * b + q] for p in (0, 1) for q in (0, 1)) / 4 for a in range(6) for b in range(6)] def nearest(store, data): d = 0 for x, y in data: t = min(store, key=lambda u: sum((u[0][i] - x[i]) ** 2 for i in range(len(x)))) d += 1 if t[1] == y else 0 return d / len(data) TEST_F = [(features(g), e) for g, _, e in TEST] print(f"\nwith forty labels, two separate slices of forty (most frequent class {MAJORITY:.4f})") print(f"{'setup':<36}{'examples':>9}{'slice 1':>9}{'slice 2':>9}") for name, choice in (("no augmentation", ()), ("+ shift", ("shift (1,1)",)), ("+ two flips", ("horizontal flip", "vertical flip")), ("+ quarter turn, label kept", ("quarter turn",)), ("+ quarter turn, label corrected", ("quarter turn*",))): count, results = 0, [] for start in (0, 40): FEW = TRAIN[start:start + 40] store = [(features(g), e) for g, _, e in FEW] for s in choice: f = dict(TRANSFORMS)[s.rstrip("*")] store += [(features(f(g)), (1 - e) if s.endswith("*") else e) for g, _, e in FEW] count = len(store) results.append(nearest(store, TEST_F)) print(f"{name:<36}{count:>9}{results[0]:>9.4f}{results[1]:>9.4f}") ``` ``` transform label broken shift (1,1) 1/160 horizontal flip 0/160 vertical flip 0/160 quarter turn 160/160 brightness +25 0/160 with forty labels, two separate slices of forty (most frequent class 0.5312) setup examples slice 1 slice 2 no augmentation 40 0.5938 0.6250 + shift 80 0.6625 0.6687 + two flips 120 0.6875 0.6250 + quarter turn, label kept 80 0.5188 0.5000 + quarter turn, label corrected 80 0.6312 0.7063 ``` The upper table is this lesson's image-specific number. Four of the five transforms leave the label as it is: flips mirror the block and do not change its dimensions; brightness does not touch the geometry; the shift breaks it in only **1** example, the one where the block overflows the edge and is clipped. The quarter turn, on the other hand, flips the label in **160 of 160 examples**, because it swaps height and width, and the label is exactly the comparison of those two. Augmentation has no task-independent list: the same transform is harmless in a task whose label is the block's **position** and destructive in a task whose label is the block's **orientation**. The lower table counts the cost. With forty labels, nearest neighbor reads **0.5938** and **0.6250** without augmentation. When two flips are added it reads 0.6875 in the first slice and 0.6250 in the second; when the shift is added, 0.6625 and 0.6687. The contribution rises to 0.0937 in the first slice and stays at 0.0437 in the second: the difference between the two slices is the same size as the difference augmentation adds. In the training practice lesson of the Deep Learning course the same contribution had been measured as +0.0026, +0.0013, and −0.0079; it shrinks as the scale grows. When the transform that breaks the label is added, the number becomes **0.5188** and **0.5000**: below the case without augmentation, and below even the 0.5312 of the most frequent class. When the same images are added with their labels corrected, 0.6312 and 0.7063 are read; the only difference between the two rows is what was written as the label. **Where** the transforms are applied is also critical, and here they are applied only to the training slice. The Deep Learning course had measured that doing augmentation before the split shifted the held-out set figure by +0.0203; that leak is not repeated here. #### Summary - An image is 12x12 pixels and one channel, that is 144 numbers; with three channels it would hold 432. The image means wander between 32.9 and 108.0. - The domain baseline is two procedures: the most frequent class is 0.5312, the mean image is 0.5500. The mean image cannot read orientation, because the orientation is erased when the position is averaged. - Dataset-wide scaling moves no number (0.5500 and 0.6937 stay as they are); this step is not a preprocessing but a change of units. - Per-image normalization raises the threshold rule from 0.6937 to 0.9250; the difference is 0.2313 and comes entirely from the representation. The same representation lowers the mean image from 0.5500 to 0.4875. - The quarter turn flips the label in 160 of 160 examples, the shift breaks it in 1, and the flips and brightness break it in none. - With forty labels, 0.5938 and 0.6250 are read without augmentation; two flips give 0.6875 and 0.6250, and a quarter turn with its label kept gives 0.5188 and 0.5000 — below the baseline. #### Next Step In this lesson the deciding procedures either did not learn at all (the threshold rule) or stored the training set as it was (nearest neighbor). The 0.9250 that the threshold rule reads on the normalized representation is the number of three lines written by a human who **knows** the geometry of the task. The next lesson removes those three lines, puts a learning model in their place, and measures two things: how far the model exceeds the domain baseline, and, when the number of labels drops to twenty, how many examples a representation carried over from another task is worth. ### Image Classification Source: https://academia.sh/en/courses/nlp-and-vision/image-classification Measuring end-to-end training and transfer learning against the domain baseline: while the most frequent class reads 0.5312 and the mean image 0.5500, the network with 1,169 parameters reads 0.6000 in both runs on 144 pixels, and the network with 305 parameters reads 0.6687 and 0.7063 on 36 pooled numbers; both memorize the training set completely. With twenty labels, training from scratch stays below the domain baseline at 0.5188 and 0.5250, while freezing a pretrained model produced from a fictional source task gives 0.6312 and 0.5500, and fine-tuning gives 0.5938 and 0.5625. At sixty labels the advantage closes, and at one hundred sixty training from scratch pulls ahead with 0.6125 and 0.6875; the gap of 0.0812 between the two runs of freezing is larger than the difference transfer adds. The highest number of the previous lesson did not come from a model. On the per-image normalized representation, a rule of three lines that extracts the bounding box of the above-threshold pixels and compares its height with its width read 0.9250 on the test set. Writing that rule required knowing the geometry of the task: that the label is the block's orientation, that the block is single, that the noise stays below the threshold. This lesson takes that knowledge back. The rule is deleted, a model that learns from labeled examples is put in its place, and two questions are asked. The first is how far the model exceeds the domain baseline. The second is what happens when the number of labels drops to twenty — and how many labeled examples the representation carried by a model trained **on another task** is worth. - **CV9. The image fiction is the same as in the previous lesson:** 12x12 single-channel gray-level image, a single rectangular block, the label is the block's orientation, seed 20260218, split 160/80/160. - **CV10. The domain baseline is two procedures:** the most frequent class and the **mean image**. The second is measured on raw pixels and reads 0.5500 on this set. - **CV11. The model is a small network with one hidden layer** and is written inside the lesson. Eight hidden units, sigmoid output, update per example, one hundred epochs. - **CV12. Every setup is run twice.** In the second run only the initial values and the shuffling order change; the data, the split, the epoch count, and the learning step are the same. - **CV13. Pooling and convolution are not built here.** 2x2 mean pooling is an operation measured in the Deep Learning course, and in this lesson it is used only as an **input representation**. - **CV14. The source task is fiction and separate from the target task:** another 240 images, the question "is the block wide". The pretrained model is produced from this task **by our own hands**; it sees no label and no image of the target task. - **CV15. Freezing** is taking the hidden layer from the source and leaving it unchanged; **fine-tuning** is continuing to train the same layer on the target task. In both, the output layer is rebuilt. #### The Domain Baseline and the Representation ```python # data.py — MODEL: The image fiction of the previous lesson continues unchanged: a 12x12 # single-channel gray-level image, a single rectangular block, the label is the block's orientation. SEED, M32, SIZE = 20260218, 0xFFFFFFFF, 12 MIN_SIDE, MAX_SIDE, HID = 2, 6, 8 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def image(no): r = rng(SEED + 700000 + no) background, contrast = 30 + int(r() * 70), 20 + int(r() * 36) height = MIN_SIDE + int(r() * (MAX_SIDE - MIN_SIDE + 1)) width = MIN_SIDE + int(r() * (MAX_SIDE - MIN_SIDE + 1)) while width == height: width = MIN_SIDE + int(r() * (MAX_SIDE - MIN_SIDE + 1)) i = int((r() + r() + r()) / 3 * (SIZE - height + 1)) j = int((r() + r() + r()) / 3 * (SIZE - width + 1)) g = [[0] * SIZE for _ in range(SIZE)] for a in range(SIZE): for b in range(SIZE): v = background + (contrast if i <= a < i + height and j <= b < j + width else 0) v += int(r() * 61) - 30 g[a][b] = 0 if v < 0 else (255 if v > 255 else v) return g, (i, j, height, width), (1 if width > height else 0) def shuffle(n, seed): r, s = rng(seed), list(range(n)) for i in range(n - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return s RAW = [image(i) for i in range(400)] DATA = [RAW[i] for i in shuffle(400, SEED + 40000)] TRAIN, VAL, TEST = DATA[:160], DATA[160:240], DATA[240:] def flat(g): return [v for s in g for v in s] def norm_t(g): # per-image normalization x = flat(g) m = sum(x) / len(x) sd = (sum((v - m) ** 2 for v in x) / len(x)) ** 0.5 return [(v - m) / sd for v in x] def features(g): # normalize, 2x2 mean pool x = norm_t(g) return [sum(x[(2 * a + p) * SIZE + 2 * b + q] for p in (0, 1) for q in (0, 1)) / 4 for a in range(6) for b in range(6)] def mean_image(train, test): """Domain baseline: class means, the decision is the nearest mean.""" M = {} for e in (0, 1): group = [x for x, y in train if y == e] M[e] = [sum(a[i] for a in group) / len(group) for i in range(len(group[0]))] def dist(x, e): return sum((x[i] - M[e][i]) ** 2 for i in range(len(x))) return sum(1 for x, y in test if (1 if dist(x, 1) < dist(x, 0) else 0) == y) / len(test) MAJORITY = max(sum(e for _, _, e in TEST), len(TEST) - sum(e for _, _, e in TEST)) / len(TEST) BASELINE = mean_image([(flat(g), e) for g, _, e in TRAIN], [(flat(g), e) for g, _, e in TEST]) print(f"split {len(TRAIN)}/{len(VAL)}/{len(TEST)}, input {SIZE * SIZE} pixels; " f"after 2x2 pooling {len(features(TRAIN[0][0]))} numbers") print(f"domain baseline, most frequent class: {MAJORITY:.4f}") print(f"domain baseline, mean image: {BASELINE:.4f}") ``` ``` split 160/80/160, input 144 pixels; after 2x2 pooling 36 numbers domain baseline, most frequent class: 0.5312 domain baseline, mean image: 0.5500 ``` The mean image procedure keeps the mean image of the two classes and writes a new image to whichever is closer. It stores 288 numbers in total, makes no iterations, and asks for no tuning variable. It is the dumbest procedure of the domain and reads **0.5500**; 0.0188 higher than the 0.5312 of the most frequent class. Every number from here on is read together with these two. In this lesson the validation set is not looked at. The epoch count, the learning step, and the hidden unit count are held the same in all setups; no sweep is done. The reason is what the measure is: the question asked is not "which network is best" but "how much does changing the input representation or the initial weights move the number on the same budget". If the tuning budget changed between setups, it could not be read whether the difference came from the tuning or from the representation. #### End-to-End Training ```python # network.py — MODEL: A small network with one hidden layer, written inside the lesson. # Sigmoid output, update per example, cross-entropy loss. import math def build(D, seed): r, a = rng(seed), (1 / D) ** 0.5 return [[[(r() * 2 - 1) * a for _ in range(D)] for _ in range(HID)], [0.0] * HID, [(r() * 2 - 1) * (1 / HID) ** 0.5 for _ in range(HID)], [0.0]] def forward(P, x): W, b, v, c = P h = [max(0.0, b[k] + sum(W[k][i] * x[i] for i in range(len(x)))) for k in range(HID)] z = c[0] + sum(v[k] * h[k] for k in range(HID)) return h, (1 / (1 + math.exp(-z)) if z > -30 else 0.0) def train_model(P, data, epochs, step, seed, hidden=True): W, b, v, c = P r, n, D = rng(seed), len(data), len(data[0][0]) for _ in range(epochs): s = list(range(n)) for i in range(n - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] for idx in s: x, y = data[idx] h, p = forward(P, x) for k in range(HID): if hidden and h[k] > 0.0: dh = (p - y) * v[k] b[k] -= step * dh for i in range(D): W[k][i] -= step * dh * x[i] v[k] -= step * (p - y) * h[k] c[0] -= step * (p - y) return P def accuracy(P, data): return sum(1 for x, y in data if (forward(P, x)[1] >= 0.5) == (y == 1)) / len(data) def dataset(transform, data): return [(transform(g), e) for g, _, e in data] print(f"{'input':<22}{'parameters':>11}{'run':>7}{'train':>8}{'test':>8}") print(f"{'most frequent class':<22}{0:>11}{'-':>7}{'-':>8}{MAJORITY:>8.4f}") print(f"{'mean image':<22}{2 * SIZE * SIZE:>11}{'-':>7}{'-':>8}{BASELINE:>8.4f}") for name, transform, D in (("144 pixels", norm_t, SIZE * SIZE), ("36 numbers (pooled)", features, 36)): E, S = dataset(transform, TRAIN), dataset(transform, TEST) for t in (0, 1): P = train_model(build(D, SEED + 11 + t), E, 100, 0.05, SEED + 511 + t) print(f"{name if t == 0 else '':<22}{D * HID + 2 * HID + 1 if t == 0 else '':>11}" f"{t + 1:>7}{accuracy(P, E):>8.4f}{accuracy(P, S):>8.4f}") ``` ``` input parameters run train test most frequent class 0 - - 0.5312 mean image 288 - - 0.5500 144 pixels 1169 1 1.0000 0.6000 2 1.0000 0.6000 36 numbers (pooled) 305 1 1.0000 0.6687 2 1.0000 0.7063 ``` In all four rows the training accuracy is **1.0000**. One hundred sixty images do not resemble one another in a 144-dimensional space; the network is wide enough to memorize them one by one, and memorizing is easier than learning what the orientation is. The test numbers are the counterpart of this: **0.6000** and **0.6000**. Reducing the input from 144 numbers to 36 numbers — taking the mean of 2x2 windows — raises the test number to **0.6687** and **0.7063**. The parameter count drops from 1,169 to **305**. The difference is not an optimization gain but an assumption: pooling says "what matters is not a pixel's value on its own but its mean together with its neighbors", and this is true in this fiction, because the block is defined not by a single pixel but by an adjacent region. The gap between the two runs is **0.0376** on the pooled input. This range, opened by only the initial values changing on the same network, the same data, and the same budget, is as large as half of the 0.0875 difference pooling adds. The 0.9250 of the hand-written threshold rule is still far away, and this is not the fault of this network but of the measure itself: the rule knows the geometry of the task, while the network has to extract it from one hundred sixty examples. #### Transfer Learning ```python # transfer.py — MODEL: The pretrained model is produced by ourselves from a FICTIONAL # source task: the source task is the question "is the block wide", it sees no label of # the target task and uses none of the target images. SOURCE = [image(1000 + i) for i in range(240)] SOURCE_DATA = [(features(g), 1 if k[3] >= 4 else 0) for g, k, _ in SOURCE] PRETRAINED = [train_model(build(36, SEED + 21 + t), SOURCE_DATA, 100, 0.05, SEED + 521 + t) for t in (0, 1)] print(f"source task: {len(SOURCE_DATA)} images, share of label 1 " f"{sum(y for _, y in SOURCE_DATA) / len(SOURCE_DATA):.4f}, accuracy on the source task " f"{accuracy(PRETRAINED[0], SOURCE_DATA):.4f} and {accuracy(PRETRAINED[1], SOURCE_DATA):.4f}") def transfer(P, seed): """The hidden layer is taken from the source; the output layer is rebuilt.""" r, (W, b, _, _) = rng(seed), P return [[s[:] for s in W], b[:], [(r() * 2 - 1) * (1 / HID) ** 0.5 for _ in range(HID)], [0.0]] TRAIN36, TEST36 = dataset(features, TRAIN), dataset(features, TEST) print(f"\n{'labels':>7}{'from scratch':>19}{'frozen':>19}{'fine-tuning':>19}") print(f"{'':>7}{'run 1 run 2':>19}{'run 1 run 2':>19}{'run 1 run 2':>19}") for n in (20, 60, 160): line = f"{n:>7}" for setup in ("from scratch", "frozen", "fine-tuning"): for t in (0, 1): if setup == "from scratch": P = build(36, SEED + 31 + t) else: P = transfer(PRETRAINED[t], SEED + 41 + t) P = train_model(P, TRAIN36[:n], 100, 0.05, SEED + 531 + t, hidden=(setup != "frozen")) line += f"{accuracy(P, TEST36):>9.4f}" if t else f"{accuracy(P, TEST36):>10.4f}" print(line) print(f"domain baseline: most frequent class {MAJORITY:.4f}, mean image {BASELINE:.4f}") ``` ``` source task: 240 images, share of label 1 0.6292, accuracy on the source task 1.0000 and 0.9958 labels from scratch frozen fine-tuning run 1 run 2 run 1 run 2 run 1 run 2 20 0.5188 0.5250 0.6312 0.5500 0.5938 0.5625 60 0.6188 0.6625 0.5938 0.6250 0.6188 0.6250 160 0.6125 0.6875 0.6000 0.5625 0.6188 0.6562 domain baseline: most frequent class 0.5312, mean image 0.5500 ``` The pretrained model here is not downloaded from a repository; it is produced from a fictional source task. The source task is the question "is the block wide" on 240 images different from the target's, and this task's labels are plentiful. What the hidden layer learns there is to read the block's **horizontal extent**. With twenty labels, training from scratch reads **0.5188** and **0.5250**: **below** the 0.5312 of the most frequent class and the 0.5500 of the mean image. With the same twenty labels, transfer gives 0.6312 and 0.5500 in freezing, and 0.5938 and 0.5625 in fine-tuning. Over the means, the difference transfer adds is **+0.0687** in freezing and **+0.0562** in fine-tuning. At sixty labels the advantage closes (from scratch 0.6406, freezing 0.6094, fine-tuning 0.6219), and at one hundred sixty training from scratch pulls ahead with **0.6500** while freezing stays at 0.5812. The second half of the question — with how many examples — is read from these three rows: the advantage of transfer runs out between twenty and sixty labels. A span of forty labels is the entire value of a pretrained model in this fiction. The reason lies in what the source task teaches. The question "is the block wide" teaches the hidden layer to read the block's **horizontal** extent; the target task asks whether the horizontal extent is greater than the **vertical** extent. The source carries one of the two required measurements and does not carry the comparison. A frozen layer stands still as data grows, because what it can learn is frozen in the source task. In the transfer learning lesson of the Deep Learning course the same pattern had been measured in a table: with a baseline of 0.6444, training from scratch with twenty labels gave 0.7194, freezing 0.8417, and fine-tuning 0.8083; and freezing had stood flat in all four measurements. The pattern measured on images is the same; the differences are smaller. One number limits the reading of this table. The two runs of freezing at twenty labels are 0.6312 and 0.5500, a gap of **0.0812**. This is **larger** than the 0.0687 difference transfer adds. Writing the number of a single run would have shown transfer stronger or weaker than it is; two runs say that the sign of the difference can be trusted but its magnitude cannot. #### Summary - The domain baseline on this task is the most frequent class at 0.5312 and the mean image at 0.5500; the second stores 288 numbers and makes no iterations. - The network with 1,169 parameters reads 0.6000 in both runs on 144 pixels; the network with 305 parameters reads 0.6687 and 0.7063 on 36 pooled numbers. All four memorize the training set at 1.0000. - The 0.0875 difference pooling adds comes from the representation; the gap between the two runs of the same network is 0.0376, which is as large as half of that difference. - With twenty labels, training from scratch stays below the domain baseline at 0.5188 and 0.5250; freezing a model produced from a fictional source task adds +0.0687, and fine-tuning adds +0.0562. - At sixty labels the advantage closes, at one hundred sixty training from scratch pulls ahead at 0.6500, and freezing stays at 0.5812. - The gap of 0.0812 between the two runs of freezing at twenty labels is larger than the difference transfer adds. #### Next Step In these two lessons the question was always the same: **what is** in the image. The answer was a single number, and the model did not have to find the block, only state its orientation. Yet a box stands inside the fiction: for every image the block's row, column, height, and width are written. The next lesson changes the question — **where** is the object. This requires a new measure, because accuracy cannot say how well two boxes match. And the domain baseline changes too: what a procedure that states the same box for every image without looking scores on that measure is the most surprising number of this course. ### Object Detection Source: https://academia.sh/en/courses/nlp-and-vision/object-detection Reading intersection over union and its threshold together with a procedure that looks at no image: a fixed 4x4 box placed in the middle of the grid gets a mean overlap of 0.3011 and counts as a detection in 0.5125 of the test images at a threshold of 0.3, but falls to 0.1938 at a threshold of 0.5 and to 0.0125 at 0.7. A fixed-size window with 81 candidates that searches only position reads 0.9750, 0.6875, and 0.2375 with a mean of 0.5687; a window with 2,025 candidates that also searches size reads 0.9812, 0.9625, and 0.8500 with a mean of 0.9076. The two procedures are separated by 0.0062 at a threshold of 0.3, and the gap between them rises to 0.6125 at a threshold of 0.7. When the confidence threshold is pulled to the score of the first quarter of the validation set, precision rises from 0.9812 to 1.0000 and recall drops to 0.7063. In two lessons the question was always the same: **what is** in the image. The answer was a single number, and the model did not need to find the block, only to say which way it points. Yet a box sits inside the fiction: for every image the block's row, column, height, and width are written down. This lesson changes the question — **where** is the object. When the question changes, the metric changes too. Accuracy cannot say how well two boxes match, because the answer is not right or wrong but **partly right**: the predicted box may have missed the ground-truth box by one cell or may have covered half of it. The first job of this lesson is to build the metric that turns this partiality into a single number, and the second is to show how misleading that number is **on its own**. - **CV16. The image fiction has not changed;** what changes is the part that is used. In place of the label, each image's **ground-truth box** is read: row, column, height, width. - **CV17. Every image has exactly one object.** This makes the task a pure **localization** problem; **detection** is localization plus the decision "is there an object, should a box be published," and it is added in the last section. - **CV18. The metric is intersection over union:** the area of the intersection of two **bounding boxes** divided by the area of their union. Zero is no contact at all, one is an exact overlap. Its direction is **bigger is better**. - **CV19. Success cannot be counted until an overlap threshold is chosen.** The threshold is read separately at 0.3, 0.5, and 0.7, and all three are printed. - **CV20. The domain baseline is the fixed box:** a single box of the mean box size in the training set, placed in the middle of the grid, stated as is for every image. This procedure **never looks at the image**. - **CV21. Precision and recall were defined in the Supervised Learning course** and are not repeated. The only thing added in this lesson is the **overlap threshold** required for a prediction to count as correct. - **CV22. Confidence threshold candidates are read from the scores of the validation set;** the test set is used only for reporting. #### Intersection over Union and the Fixed Box ```python # box.py — MODEL: Same image fiction; in this lesson not the label but each # image's GROUND-TRUTH BOX is used: (row, column, height, width). import statistics SEED, M32, SIZE = 20260218, 0xFFFFFFFF, 12 MIN_SIDE, MAX_SIDE = 2, 6 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def image(no): r = rng(SEED + 700000 + no) background, contrast = 30 + int(r() * 70), 20 + int(r() * 36) height = MIN_SIDE + int(r() * (MAX_SIDE - MIN_SIDE + 1)) width = MIN_SIDE + int(r() * (MAX_SIDE - MIN_SIDE + 1)) while width == height: width = MIN_SIDE + int(r() * (MAX_SIDE - MIN_SIDE + 1)) i = int((r() + r() + r()) / 3 * (SIZE - height + 1)) j = int((r() + r() + r()) / 3 * (SIZE - width + 1)) g = [[0] * SIZE for _ in range(SIZE)] for a in range(SIZE): for b in range(SIZE): v = background + (contrast if i <= a < i + height and j <= b < j + width else 0) v += int(r() * 61) - 30 g[a][b] = 0 if v < 0 else (255 if v > 255 else v) return g, (i, j, height, width), (1 if width > height else 0) def shuffle(n, seed): r, s = rng(seed), list(range(n)) for i in range(n - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return s RAW = [image(i) for i in range(400)] DATA = [RAW[i] for i in shuffle(400, SEED + 40000)] TRAIN, VAL, TEST = DATA[:160], DATA[160:240], DATA[240:] def overlap(a, b): """Intersection over union: intersection area divided by union area.""" di = max(0, min(a[0] + a[2], b[0] + b[2]) - max(a[0], b[0])) dj = max(0, min(a[1] + a[3], b[1] + b[3]) - max(a[1], b[1])) inter = di * dj return inter / (a[2] * a[3] + b[2] * b[3] - inter) def inside(k, a, b): return k[0] <= a < k[0] + k[2] and k[1] <= b < k[1] + k[3] sy = round(statistics.mean(k[2] for _, k, _ in TRAIN)) sx = round(statistics.mean(k[3] for _, k, _ in TRAIN)) FIXED = ((SIZE - sy) // 2, (SIZE - sx) // 2, sy, sx) THRESHOLDS = (0.3, 0.5, 0.7) def report(name, boxes): o = [overlap(k, g) for k, (_, g, _) in zip(boxes, TEST)] print(f"{name:<32}{statistics.mean(o):>13.4f}" + "".join(f"{sum(1 for v in o if v >= e) / len(o):>10.4f}" for e in THRESHOLDS)) return o g0, k0, _ = TEST[0] print(f"mean box in the training set {sy}x{sx}; fixed box {FIXED}") print(f"example: ground-truth box {k0}; X intersection, g ground truth only, f fixed only") for a in range(SIZE): print(" " + "".join("Xgf."[(0 if inside(FIXED, a, b) else 1) if inside(k0, a, b) else (2 if inside(FIXED, a, b) else 3)] for b in range(SIZE))) print(f"in this example the intersection over union is {overlap(FIXED, k0):.4f}") print(f"\n{'domain baseline':<32}{'mean overlap':>13}" + "".join(f"{'thr ' + str(e):>10}" for e in THRESHOLDS)) FIXED_O = report("fixed box (ignores the image)", [FIXED] * len(TEST)) ``` ``` mean box in the training set 4x4; fixed box (4, 4, 4, 4) example: ground-truth box (3, 6, 6, 5); X intersection, g ground truth only, f fixed only ............ ............ ............ ......ggggg. ....ffXXggg. ....ffXXggg. ....ffXXggg. ....ffXXggg. ......ggggg. ............ ............ ............ in this example the intersection over union is 0.2105 domain baseline mean overlap thr 0.3 thr 0.5 thr 0.7 fixed box (ignores the image) 0.3011 0.5125 0.1938 0.0125 ``` The drawing shows the definition of the metric cell by cell. The intersection is 8 cells, and the union is 38 cells: the 30 cells of the ground-truth box plus the 16 cells of the fixed box, minus the intersection. The ratio is 8/38 = 0.2105. This is a single example, and the boxes are quite far apart. The bottom row is where the second claim of this course is paid off. The fixed box looks at no image, reads no pixel, learns nothing: the only information it takes from the training set is the mean box size, and its position is the middle of the grid. This procedure gets a mean overlap of **0.3011** and **counts as a detection in 0.5125 of the test images at a threshold of 0.3** — in more than half of them. The source of the number is not the metric but the fiction, and the fiction carries it on purpose. Because the block's position is chosen by the mean of three independent draws, the objects gather in the middle of the grid. In real image sets too the object is mostly in the middle, and this is enough for a procedure that has learned nothing to look good on the overlap metric. When the threshold is raised, the same procedure falls to **0.1938** and **0.0125**: the same box, the same test set, and the only thing that changed is the reading threshold. #### What the Search Adds ```python # window.py — MODEL: Sliding window: the score of each candidate box is the sum of the # normalized pixels inside it divided by the square root of the area. # The sums are read in constant time from the cumulative sum table. def norm_t(g): x = [v for s in g for v in s] m = sum(x) / len(x) sd = (sum((v - m) ** 2 for v in x) / len(x)) ** 0.5 return [[(v - m) / sd for v in s] for s in g] def sum_table(x): T = [[0.0] * (SIZE + 1) for _ in range(SIZE + 1)] for a in range(SIZE): for b in range(SIZE): T[a + 1][b + 1] = x[a][b] + T[a][b + 1] + T[a + 1][b] - T[a][b] return T def search(x, sizes): T, best, box = sum_table(x), None, FIXED for h, w in sizes: root = (h * w) ** 0.5 for a in range(SIZE - h + 1): for b in range(SIZE - w + 1): p = (T[a + h][b + w] - T[a][b + w] - T[a + h][b] + T[a][b]) / root if best is None or p > best: best, box = p, (a, b, h, w) return box, best ONE = [(FIXED[2], FIXED[3])] ALL = [(h, w) for h in range(MIN_SIDE, MAX_SIDE + 1) for w in range(MIN_SIDE, MAX_SIDE + 1)] TEST_N = [norm_t(g) for g, _, _ in TEST] def candidate_count(sizes): return sum((SIZE - h + 1) * (SIZE - w + 1) for h, w in sizes) print(f"{'procedure':<32}{'mean overlap':>13}" + "".join(f"{'thr ' + str(e):>10}" for e in THRESHOLDS)) report("fixed box (ignores the image)", [FIXED] * len(TEST)) report("fixed-size sliding window", [search(x, ONE)[0] for x in TEST_N]) FULL = [search(x, ALL) for x in TEST_N] report("window that also searches size", [k for k, _ in FULL]) print(f"candidate boxes: fixed-size {candidate_count(ONE)}, " f"also searching size {candidate_count(ALL)}") ``` ``` procedure mean overlap thr 0.3 thr 0.5 thr 0.7 fixed box (ignores the image) 0.3011 0.5125 0.1938 0.0125 fixed-size sliding window 0.5687 0.9750 0.6875 0.2375 window that also searches size 0.9076 0.9812 0.9625 0.8500 candidate boxes: fixed-size 81, also searching size 2025 ``` The only difference between the two procedures is what is searched. The fixed-size window searches only **position**: the box's height is fixed to the mean of the training set, and 81 positions are tried. The second also searches **size** and evaluates 2,025 candidates. The mean overlap rises from 0.5687 to **0.9076**; a search space 25 times larger adds 0.3389. The real number is not in the rows but in the gap between the columns. **At a threshold of 0.3** the two windows read 0.9750 and 0.9812; the gap between them is **0.0062**, which is to say it can be ignored. **At a threshold of 0.7** the same two procedures read 0.2375 and 0.8500; the gap is **0.6125**. The same two procedures, the same test set, the same family of metrics — only the threshold changes, and one becomes "nearly equal" and the other "three and a half times better." What the threshold means also changes from procedure to procedure. The fixed-size window finds the object but cannot match its size: when a 4x4 box standing at the right position meets a 6x5 ground-truth box, the intersection can be at most 16 cells and the union stays at 30 cells, so the overlap cannot exceed 16/30 = 0.5333 and **structurally** cannot pass the 0.7 threshold. A low threshold hides this flaw entirely. An overlap number cannot be carried around without writing at which threshold it was read. The form of the score is also a decision, and it has a measurable consequence. If the raw pixel sum of the candidate boxes were used, the search would pick the largest box, because even though the background pixels are zero on average, the sum is inflated by noise that grows with the size of the box; if the **mean** inside the box were used, the search would pick a single bright noise cell, because the smallest box gives the highest mean. Dividing the sum by the square root of the area balances the two tendencies: the score measures how far the region stands out against the noise. #### Confidence Threshold, Precision, and Recall ```python # confidence.py — Until now every procedure published one box per image. When # no box is published while the score is below a confidence threshold, precision # and recall separate. Threshold candidates are read from the scores of the # VALIDATION set. BOXES, SCORES = [k for k, _ in FULL], [p for _, p in FULL] VAL_S = sorted(search(norm_t(g), ALL)[1] for g, _, _ in VAL) CANDIDATES = [("all", 0.0)] + [(f"quartile {q}", VAL_S[len(VAL_S) * q // 4]) for q in (1, 2, 3)] def precision_recall(confidence, threshold): published = [(k, g) for k, p, (_, g, _) in zip(BOXES, SCORES, TEST) if p >= confidence] correct = sum(1 for k, g in published if overlap(k, g) >= threshold) return (correct / len(published) if published else 0.0), correct / len(TEST) print(f"{'confidence threshold':<22}{'published':>10}" + "".join(f"{f'{e} prec':>9}{f'{e} rec':>9}" for e in THRESHOLDS)) for name, confidence in CANDIDATES: share = sum(1 for p in SCORES if p >= confidence) / len(SCORES) print(f"{name:<22}{share:>10.4f}" + "".join(f"{p:>9.4f}{r:>9.4f}" for p, r in (precision_recall(confidence, e) for e in THRESHOLDS))) print(f"{'fixed box':<22}{1.0:>10.4f}" + "".join(f"{v:>9.4f}{v:>9.4f}" for v in (sum(1 for o in FIXED_O if o >= e) / len(FIXED_O) for e in THRESHOLDS))) ``` ``` confidence threshold published 0.3 prec 0.3 rec 0.5 prec 0.5 rec 0.7 prec 0.7 rec all 1.0000 0.9812 0.9812 0.9625 0.9625 0.8500 0.8500 quartile 1 0.7063 1.0000 0.7063 1.0000 0.7063 0.9735 0.6875 quartile 2 0.4437 1.0000 0.4437 1.0000 0.4437 1.0000 0.4437 quartile 3 0.2625 1.0000 0.2625 1.0000 0.2625 1.0000 0.2625 fixed box 1.0000 0.5125 0.5125 0.1938 0.1938 0.0125 0.0125 ``` Until now every procedure published a box in every image; because it had to publish, precision and recall were the same number. The first row shows this: at a publish rate of 1.0000 both are 0.9812. When predictions with a low score are not published, the two separate. When the score of the lowest quarter of the validation set is taken as the threshold, a box is published in 0.7063 of the images, and at a threshold of 0.3 precision rises to **1.0000** while recall drops to **0.7063**. The Supervised Learning course set up this trade-off and its definitions are not repeated here. The only thing specific to images is this: there a prediction was "right" or "wrong," while here an **overlap threshold** defines being right, and that threshold separates the columns of the table. At the quarter threshold the precision in the 0.7 column stays at 0.9735, because even a prediction with a high score can miss a 0.7 overlap. The score does not say that the box is right; it says that a bright cluster was found in that region. The last row puts the baseline into the same table. Because the fixed box always publishes, its precision and recall are equal, and at a threshold of 0.3 they are 0.5125. Looking at the precision and saying "my procedure is 1.0000" says nothing unless it is written at which threshold and in how many images a box was published; and without the domain baseline written down, it counts as unmeasured. #### Summary - Intersection over union is the area of the intersection of two boxes divided by the area of their union; in the example the intersection is 8 cells, the union is 38 cells, and the ratio is 0.2105. - The 4x4 fixed box placed in the middle of the grid gets a mean overlap of 0.3011 without looking at the image at all and counts as a detection in 0.5125 of the test images at a threshold of 0.3; its source is the center bias in the fiction. - The same fixed box falls to 0.1938 at a threshold of 0.5 and to 0.0125 at 0.7: the number comes not from the procedure but from the threshold. - The window with 81 candidates that searches only position gives a mean overlap of 0.5687, and the window with 2,025 candidates that also searches size gives 0.9076; a 25-fold search adds 0.3389. - The two windows are separated by 0.0062 at a threshold of 0.3, and the gap rises to 0.6125 at a threshold of 0.7; a low threshold entirely hides the size error. - When the confidence threshold is pulled to the score of the first quarter of the validation set, the publish rate becomes 0.7063, precision rises from 0.9812 to 1.0000, and recall drops to 0.7063. #### Next Step The box said **where** the object is, but it did not say **which pixels** are the object. Even when the window that also searches size gets a mean overlap of 0.9076, the answer it gives is a rectangle, and every pixel inside that rectangle counts as the object. In this fiction the object is already a rectangle, so most of the loss is invisible; when the edge of the object is not straight, the background pixels that fall inside the box are not counted at all. The next lesson takes the decision down to the **pixel level**, and its first job will again be to write the baseline: a procedure that assigns every pixel to the dominant class gets a surprisingly high number because of pixel imbalance. ### Segmentation Source: https://academia.sh/en/courses/nlp-and-vision/segmentation Reading the decision at the pixel level against its own baseline: because 2,193 of the 17,280 pixels in the test set are object, the baseline that assigns every pixel to the dominant class gets 0.8731 in pixel accuracy, while the object intersection over union of the same output is 0.0000. A linear model deciding from a single pixel value gives 0.9540 and 0.6412, and the same model with a 3x3 neighborhood window gives 0.9851 and 0.8843; the difference comes not from the model but from the representation. 45 of the one hundred twenty images carry adjacent blocks, and even with a perfect mask, counting connected components gets the instance count wrong in these 45 images; rectangle decomposition fixes 24 of them and stops at 21, while on the predicted mask the wrong count rises from 46 to 79. The previous lesson said **where** the object is with a box, and compared that box's intersection over union with the figure of a fixed box placed in the middle of the image on the same metric; the window that also searches size got a mean overlap of 0.9076. One thing the box does not say is left: **every pixel** inside the rectangle that was found counts as the object. A box is a coarse answer made of four numbers; the answer to the question "which pixels" is as many decisions as there are pixels. In the fiction of this lesson the objects are rectangular blocks, so a perfect box already gives a perfect mask. In this respect the fiction is **in favor of the box**, and the lesson still shows two things. The first is that when one goes down to the pixel level, the domain baseline changes and comes out **surprisingly high**. The second is that however well decisions are made at the pixel level, **where two objects touch**, semantic segmentation cannot separate the two; this is not an accuracy problem but a limit that has to do with the form of the output. - **CV31. The image is fiction.** The 12x12 coverage grid is here a small gray-level image; the grid comes from the Visualization, Reporting and Analysis at Scale course and was turned into a learning input in the Deep Learning course. Seed 20260218. - **CV32. The objects are fiction rectangular blocks.** Height and width are between 2 and 4, the block pixels sit at a higher gray level than the background, and the noise is produced by the sum of three uniform numbers; the distributions of the two levels overlap at the tails. - **CV33. An image contains one, two, or three blocks.** The second and third blocks rest against the edge of the previous one with probability 0.40; if they do not, at least one cell of gap is left between them. - **CV34. There are two labels.** The object mask is 0/1 for every pixel and is the target of **semantic segmentation**; the instance mask writes the block number into every pixel and is the target of **instance segmentation**. - **CV35. The split is 80/120** and is fixed in this image set. The training set is small on purpose. - **CV36. The domain baseline is assigning every pixel to the dominant class.** The dominant class is the background; the reason for the choice is the pixel ratio, and that ratio is printed. - **CV37. Two metrics are read side by side.** Pixel accuracy (bigger is better) counts all the pixels; the **intersection over union** of the object class (intersection divided by union, bigger is better) counts only the object pixels. Intersection over union was defined on boxes in the third lesson; here it is read on sets of pixels. - **CV38. The model is linear and fitted by least squares**; the decision threshold is fixed at 0.5. That the threshold choice moves the number was measured in the Supervised Learning course and is **not repeated**. No library is called. - **CV39. The instance count is read with two procedures:** connected components, and rectangle decomposition with the prior "objects are rectangles." #### The Fiction Image and the Domain Baseline ```python # image.py — MODEL: The image set is FICTION: the 12x12 coverage grid of the Visualization, # Reporting and Analysis at Scale course and the Deep Learning course is here a small # gray-level image, and fiction rectangular blocks are placed on it. Seed 20260218. SEED, M32, SIZE = 20260218, 0xFFFFFFFF, 12 BACKGROUND, BLOCK, NOISE = 0.30, 0.74, 0.30 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def intersects(a, b, gap): return not (a[0] + a[2] + gap <= b[0] or b[0] + b[2] + gap <= a[0] or a[1] + a[3] + gap <= b[1] or b[1] + b[3] + gap <= a[1]) def image(no): """One coverage frame: gray levels, object mask, instance mask, block count.""" r = rng(SEED + 400000 + no) blocks = [] for _ in range(1 + pick(r(), [0.24, 0.46, 0.30])): h, w = 2 + int(r() * 3), 2 + int(r() * 3) adjacent = bool(blocks) and r() < 0.40 for _ in range(24): if adjacent: # resting against the edge of the previous block a, side = blocks[int(r() * len(blocks))], int(r() * 4) sy = a[0] - h if side == 0 else a[0] + a[2] if side == 1 else a[0] sx = a[1] if side < 2 else a[1] - w if side == 2 else a[1] + a[3] else: sy, sx = int(r() * (SIZE - h + 1)), int(r() * (SIZE - w + 1)) new = (sy, sx, h, w) if not (0 <= sy <= SIZE - h and 0 <= sx <= SIZE - w): continue if any(intersects(new, b, 0 if adjacent else 1) for b in blocks): continue blocks.append(new) break g = [[BACKGROUND + (r() + r() + r() - 1.5) * NOISE for _ in range(SIZE)] for _ in range(SIZE)] instance = [[0] * SIZE for _ in range(SIZE)] for index, (sy, sx, h, w) in enumerate(blocks, 1): for i in range(sy, sy + h): for j in range(sx, sx + w): g[i][j] += BLOCK - BACKGROUND instance[i][j] = index mask = [[1 if instance[i][j] else 0 for j in range(SIZE)] for i in range(SIZE)] return g, mask, instance, len(blocks) DATA = [image(i) for i in range(200)] TRAIN, TEST = DATA[:80], DATA[80:] PIXELS = len(TEST) * SIZE * SIZE OBJECT = sum(sum(sum(s) for s in m) for _, m, _, _ in TEST) print(f"fiction: 200 images {SIZE}x{SIZE}, train {len(TRAIN)}, test {len(TEST)}; " f"{OBJECT} object pixels / {PIXELS} in the test set") print(f"baseline - assigning every pixel to the dominant class (background): " f"pixel accuracy {(PIXELS - OBJECT) / PIXELS:.4f}, object intersection over union 0.0000") g, _, o, k = DATA[97] for i in range(SIZE): print(" " + "".join(str(min(9, max(0, int(g[i][j] * 10)))) for j in range(SIZE)) + " " + "".join(".12345"[o[i][j]] for j in range(SIZE))) print(f" gray level on the left, instance number on the right; this frame has {k} blocks") ``` ``` fiction: 200 images 12x12, train 80, test 120; 2193 object pixels / 17280 in the test set baseline - assigning every pixel to the dominant class (background): pixel accuracy 0.8731, object intersection over union 0.0000 102330223332 ............ 222322797803 ......2222.. 333006674822 ......2222.. 410235775913 ......2222.. 426354786802 ......2222.. 045333776830 ......1111.. 302211787842 ......1111.. 344130859630 ......1111.. 234251999753 ......1111.. 231130223333 ............ 432153222132 ............ 332423044303 ............ gray level on the left, instance number on the right; this frame has 2 blocks ``` The first number of the lesson is in the baseline row. 2,193 of the 17,280 pixels in the test set are object; the ratio is 0.1269. The procedure that learns nothing and writes "background" into every pixel therefore gets **0.8731** in pixel accuracy. This number is not a success but the class ratio itself: because the object pixels are the minority, missing them entirely costs little. The number of the same output on the second metric is **0.0000**. Intersection over union counts only the object class; since the intersection is empty, the ratio is zero. A single output gives 0.8731 and 0.0000 on two metrics. Here the choice of metric is not a preference but the decision that determines what the number means. The printed frame also shows the second question of the course. The instance map on the right says there are **two separate blocks** between the sixth and ninth columns; in the gray levels on the left there is a single bright rectangle. Where the two blocks touch, there is no boundary at all between the pixel values. #### The Same Model, Two Representations ```python def window(g, i, j): return [g[min(SIZE - 1, max(0, a))][min(SIZE - 1, max(0, b))] for a in (i - 1, i, i + 1) for b in (j - 1, j, j + 1)] def features(g, i, j, wide): return [1.0] + (window(g, i, j) if wide else [g[i][j]]) def solve(A, b): n, x = len(b), [0.0] * len(b) M = [A[i][:] + [b[i]] for i in range(n)] for k in range(n): p = max(range(k, n), key=lambda r: abs(M[r][k])) M[k], M[p] = M[p], M[k] for r in range(k + 1, n): f = M[r][k] / M[k][k] for c in range(k, n + 1): M[r][c] -= f * M[k][c] for k in range(n - 1, -1, -1): x[k] = (M[k][n] - sum(M[k][c] * x[c] for c in range(k + 1, n))) / M[k][k] return x def train_model(dataset, wide): """Per-pixel linear decision function; fitted by least squares. MODEL.""" d = 10 if wide else 2 A, b = [[0.0] * d for _ in range(d)], [0.0] * d for g, m, _, _ in dataset: for i in range(SIZE): for j in range(SIZE): x, y = features(g, i, j, wide), float(m[i][j]) for p in range(d): b[p] += x[p] * y for q in range(p, d): A[p][q] += x[p] * x[q] for p in range(d): for q in range(p): A[p][q] = A[q][p] return solve(A, b) def predict(w, g, wide): return [[1 if sum(a * c for a, c in zip(w, features(g, i, j, wide))) >= 0.5 else 0 for j in range(SIZE)] for i in range(SIZE)] def measure(dataset, w, wide): correct = inter = union = 0 for g, m, _, _ in dataset: k = predict(w, g, wide) if w else [[0] * SIZE for _ in range(SIZE)] for i in range(SIZE): for j in range(SIZE): correct += k[i][j] == m[i][j] inter += k[i][j] and m[i][j] union += k[i][j] or m[i][j] return correct / (len(dataset) * SIZE * SIZE), inter / max(1, union) W1, W9 = train_model(TRAIN, False), train_model(TRAIN, True) print(f"{'representation':<25}{'pixel accuracy':>16}{'intersection over union':>25}") for name, w, wide in (("baseline (all background)", None, False), ("single pixel value", W1, False), ("3x3 neighborhood window", W9, True)): acc, iou = measure(TEST, w, wide) print(f"{name:<25}{acc:>16.4f}{iou:>25.4f}") ``` ``` representation pixel accuracy intersection over union baseline (all background) 0.8731 0.0000 single pixel value 0.9540 0.6412 3x3 neighborhood window 0.9851 0.8843 ``` The only difference between the two rows is the **representation**. The model family is the same (a linear decision function), the fitting procedure is the same (least squares), the threshold is the same (0.5), the training set is the same. What changes is what the input is: a single number, or the pixel itself together with its eight neighbors. The difference shows up in two sizes on the two metrics. Pixel accuracy goes from 0.9540 to 0.9851, a movement of **0.0311**. Intersection over union goes from 0.6412 to 0.8843, a movement of **0.2431**. What the second representation adds is the same thing; the first metric shows an eighth of it, because 87% of the accuracy denominator is background pixels anyway, and both representations hardly ever err on a background pixel. Read against the baseline, the gain of the 3x3 window in pixel accuracy is 0.1120. When 0.8731 is not written next to this number, the sentence "98.5% accuracy" can be made, and that sentence hides that the procedure that learns nothing gets 0.8731. The rule of the course is paid once more here: **a number whose domain baseline is not written counts as unmeasured.** The answer to the question of where the contribution of the neighborhood window comes from is in the noise. A single pixel value errs on its own because the distributions of the two gray levels overlap at the tails; the mean of the neighbors shrinks the same noise, and a lone bright pixel in the background is suppressed by its surroundings. This is something different from the weight sharing of the convolution layer measured in the Deep Learning course: here there is a single small set of learned weights, and the gain comes from **the coverage of the input**. #### Where Semantic Segmentation Stops ```python def connected_components(mask): seen, found = [[0] * SIZE for _ in range(SIZE)], [] for i in range(SIZE): for j in range(SIZE): if mask[i][j] and not seen[i][j]: seen[i][j], stack, cells = 1, [(i, j)], [] while stack: a, b = stack.pop() cells.append((a, b)) for da, db in ((1, 0), (-1, 0), (0, 1), (0, -1)): p, q = a + da, b + db if 0 <= p < SIZE and 0 <= q < SIZE and mask[p][q] and not seen[p][q]: seen[p][q] = 1 stack.append((p, q)) found.append(cells) return found def rectangle_count(cells): """With the prior that objects are rectangles, how many rectangles a component splits into.""" counts = [] for axis in (0, 1): profile, pieces, previous = {}, 0, None for a, b in cells: profile.setdefault(a if axis == 0 else b, set()).add(b if axis == 0 else a) for k in sorted(profile): if previous is None or k != previous + 1 or profile[k] != profile[previous]: pieces += 1 previous = k counts.append(pieces) return min(counts) def tally(dataset, w): adjacent = component = rectangle = 0 for g, m, _, k in dataset: mask = predict(w, g, True) if w else m b = connected_components(mask) adjacent += len(connected_components(m)) < k component += len(b) != k rectangle += sum(rectangle_count(h) for h in b) != k return adjacent, component, rectangle for name, w in (("ground-truth mask", None), ("3x3 prediction", W9)): adjacent, component, rectangle = tally(TEST, w) print(f"{name:<18} carrying adjacent blocks {adjacent:>3}/{len(TEST)} images; " f"instance count wrong: by counting components {component:>3}, " f"by rectangle decomposition {rectangle:>3}") ``` ``` ground-truth mask carrying adjacent blocks 45/120 images; instance count wrong: by counting components 45, by rectangle decomposition 21 3x3 prediction carrying adjacent blocks 45/120 images; instance count wrong: by counting components 46, by rectangle decomposition 79 ``` The first row is the ceiling of semantic segmentation, and this ceiling is measured with a **perfect mask**. 45 of the one hundred twenty test images carry adjacent blocks; the procedure that counts connected components gives the instance count wrong in exactly those 45 images and right in the remaining 75. This is the ratio even when there is no error in the mask at all, because the error is not in the mask but **in the form of the question**: semantic segmentation writes one class into every pixel, and the pixels of two adjacent blocks are in the same class. The second procedure adds a prior about the shape of the object — the blocks are rectangles — and counts into at least how many rectangles the component has to be split. On the perfect mask this brings the 45 errors down to **21**: in 24 images the adjacency can be read from the non-rectangular shape of the component. In the remaining 21 images the union of the two blocks **is itself a rectangle**; the frame printed in the previous section is exactly this, and no shape prior can separate the two blocks there. The third number shows the price of the prior. On the predicted mask the error of the procedure that counts components rises from 45 to 46, which is to say the mask error does almost no harm. Rectangle decomposition, on the other hand, jumps from 21 to **79**: a single missing or extra pixel in the prediction takes the component out of being a rectangle, and the procedure splits it into two or three pieces. The prior wins on perfect input and loses on noisy input; and this happens on a mask whose intersection over union is 0.8843. This is where semantic and instance segmentation part. The output of semantic segmentation is **one class per pixel**, and a good pixel accuracy never brings this separation; the output of instance segmentation is **sets of pixels**, and for this either a separate output per object or outside information such as shape is needed. The metric parts as well: while pixel accuracy and intersection over union stay near the ceiling in 45 images, the instance count is wrong. #### Summary - Because 2,193 of the 17,280 pixels in the test set are object, the baseline that assigns every pixel to the dominant class gets 0.8731 in pixel accuracy; the object intersection over union of the same output is 0.0000. - The same linear model gives 0.9540 and 0.6412 with a single pixel value and 0.9851 and 0.8843 with a 3x3 neighborhood window; the difference comes not from the model family but from the representation, and it shows as 0.0311 and 0.2431 on the two metrics. - 45 of the one hundred twenty images carry adjacent blocks; even with a perfect mask, counting connected components gets the instance count wrong in these 45 images, because the limit is not in the mask but in the form of the question. - The prior that objects are rectangles brings the error down to 21 on the perfect mask and raises it to 79 on the predicted mask; the prior makes one pay on noisy input. #### Next Step In four lessons the image was a single frame, and every decision was read from that frame itself. A coverage network produces frames not one by one but **one after another**: successive readings of the same region form a sequence, and the neighborhood of the sequence is not in space but in time. In this lesson the neighborhood window moved pixel accuracy by 0.0311 and intersection over union by 0.2431; which number moves when the window opens into time. The next lesson counts the **inconsistency over time** of the decision given per frame, and shows what the best number is that a procedure that learns nothing can get on that metric. ### Video and Temporal Models Source: https://academia.sh/en/courses/nlp-and-vision/video-and-temporal-models Counting the instability over time of the per-frame decision: on the 432 frames of thirty-six test sequences, the baseline that writes the dominant class into every frame gives 0.7014 accuracy and 0 instability, while the true label sequences carry 71 transitions. The independent frame decision produces 99 transitions with 0.8634 accuracy, which is 28 more than the truth. When a transition penalty is applied to the same scores, accuracy rises to 0.8657 at a penalty of 0.05, instability settles on the truth's 71 at a penalty of 0.15, and at a penalty of 4.00 the decision falls to the baseline itself, 0.7014 and 1 transition: on this metric the domain baseline is not a separate procedure but the extreme value of the penalty. The previous lesson made the pixel decision by looking at neighbors, and the neighborhood window moved the intersection over union by 0.2431. That neighborhood was **in space**: the eight neighbors of a pixel were inside the same frame. A coverage network produces frames not one by one but one after another; the same region is read again at certain intervals, and the readings form a sequence. The neighborhood of this sequence is **in time**, and the question of the lesson is this: what does a procedure that decides on every frame independently lose when it never uses time. The loss is not seen by looking at frame accuracy. An event really starts, lasts for a while, and ends; the label sequence therefore changes two or three times. A procedure that decides independently on each frame changes its mind every time the noise changes direction. The second metric of this lesson is **instability**: how many times the decision changes between two consecutive frames. The metric also brings a new question, because the baseline that learns nothing gets **zero** on this metric. - **CV40. The temporal fiction is a short sequence of the frame of the fourth lesson.** The same 12x12 gray-level coverage frame is produced twelve times for one region; the background, the block level, and the noise are the same, and the seed is 20260218. - **CV41. The event is an interval.** The fiction block is present in two to five consecutive frames of the sequence; the starting frame is between the first and the seventh frame. The frame label is whether the block is present in that frame. - **CV42. There are two sources of noise.** In about a third of the active frames the reading is weak and the block looks dim; in about one in seven of the inactive frames a transient bright blob appears. The first produces a miss, the second a false alarm. - **CV43. The split is 24/36 sequences**, that is, there are 432 frames in the test set. The split is at the sequence level; the frames of the same sequence are not distributed to two sets. - **CV44. The domain baseline is writing the dominant class into every frame.** The dominant class is "not active." - **CV45. Two metrics are read side by side.** Frame accuracy (bigger is better) and **instability**: how many times the decision changes across consecutive frames. In instability, small is not good; the number to read is **the instability of the true label sequences**, and that number is printed too. - **CV46. The frame model is linear** and is fitted from three numbers per frame: the mean gray level, the mean of the brightest 3x3 window, and the ratio of above-threshold pixels. Least squares, threshold 0.5. No library is called. - **CV47. The decision that uses time uses the same scores.** The model, the weights, and the frame scores do not change; only **a fixed penalty per transition** is added to the whole sequence, and the lowest-cost decision sequence is chosen. #### The Temporal Fiction and Two Baseline Numbers ```python # sequence.py — MODEL: The temporal fiction is A SHORT SEQUENCE of the gray-level 12x12 # coverage frame of the fourth lesson: the same region is read twelve times and the # event is active in some of the frames. Seed 20260218. SEED, M32, SIZE, FRAMES = 20260218, 0xFFFFFFFF, 12, 12 BACKGROUND, BLOCK, NOISE = 0.30, 0.74, 0.30 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def sequence(no): """Twelve-frame reading of one region: frames and the event label per frame.""" r = rng(SEED + 500000 + no) start, length = 1 + int(r() * 7), 2 + int(r() * 4) h, w = 2 + int(r() * 3), 2 + int(r() * 3) sy, sx = int(r() * (SIZE - h + 1)), int(r() * (SIZE - w + 1)) frames, labels = [], [] for t in range(FRAMES): active = start <= t < start + length g = [[BACKGROUND + (r() + r() + r() - 1.5) * NOISE for _ in range(SIZE)] for _ in range(SIZE)] if active: strength = 0.35 if r() < 0.28 else 1.0 # the reading is weak in some frames for i in range(sy, sy + h): for j in range(sx, sx + w): g[i][j] += (BLOCK - BACKGROUND) * strength elif r() < 0.14: # transient bright blob a, b = int(r() * (SIZE - 1)), int(r() * (SIZE - 1)) for i in (a, a + 1): for j in (b, b + 1): g[i][j] += BLOCK - BACKGROUND frames.append(g) labels.append(int(active)) return frames, labels DATA = [sequence(i) for i in range(60)] TRAIN, TEST = DATA[:24], DATA[24:] TOTAL = len(TEST) * FRAMES ACTIVE = sum(sum(e) for _, e in TEST) BASELINE = (TOTAL - ACTIVE) / TOTAL TRUE_INSTABILITY = sum(sum(e[t] != e[t - 1] for t in range(1, FRAMES)) for _, e in TEST) print(f"fiction: {len(DATA)} sequences x {FRAMES} frames, train {len(TRAIN)}, test {len(TEST)}; " f"{ACTIVE}/{TOTAL} frames active in the test set") print(f"baseline - writing the dominant class (not active) to every frame: " f"frame accuracy {BASELINE:.4f}, instability 0") print(f"instability of the true label sequences: {TRUE_INSTABILITY} transitions " f"({TRUE_INSTABILITY / len(TEST):.2f} per sequence)") ``` ``` fiction: 60 sequences x 12 frames, train 24, test 36; 129/432 frames active in the test set baseline - writing the dominant class (not active) to every frame: frame accuracy 0.7014, instability 0 instability of the true label sequences: 71 transitions (1.97 per sequence) ``` The domain baseline gives two numbers, and the two are read in opposite directions. In frame accuracy it gets **0.7014**, because only 129 of the 432 frames are active. In instability it gets **0**, because it never changes its mind; a procedure that repeats a constant cannot be beaten on this metric. The third line says how the metric is to be read. The true label sequences carry **71 transitions** in total over 36 sequences, 1.97 per sequence. If an event starts and ends, that makes two transitions; for those that last until the last frame of the sequence, one transition remains. Small is not good in instability: the number sought is not zero but **71**. The baseline's zero is below the truth, and this is the clearest form of the fact that the metric cannot be read on its own. #### The Independent Frame Decision ```python def features(g): """Three numbers per frame: mean gray level, brightest 3x3 window, above-threshold ratio.""" total = sum(sum(s) for s in g) brightest = 0.0 for i in range(SIZE - 2): for j in range(SIZE - 2): p = sum(g[i + a][j + b] for a in range(3) for b in range(3)) / 9.0 if p > brightest: brightest = p above = sum(1 for s in g for v in s if v > 0.55) / (SIZE * SIZE) return [1.0, total / (SIZE * SIZE), brightest, above] def solve(A, b): n, x = len(b), [0.0] * len(b) M = [A[i][:] + [b[i]] for i in range(n)] for k in range(n): p = max(range(k, n), key=lambda r: abs(M[r][k])) M[k], M[p] = M[p], M[k] for r in range(k + 1, n): f = M[r][k] / M[k][k] for c in range(k, n + 1): M[r][c] -= f * M[k][c] for k in range(n - 1, -1, -1): x[k] = (M[k][n] - sum(M[k][c] * x[c] for c in range(k + 1, n))) / M[k][k] return x def train_model(dataset): """Per-frame linear decision function; fitted by least squares. MODEL.""" A, b = [[0.0] * 4 for _ in range(4)], [0.0] * 4 for frames, labels in dataset: for g, y in zip(frames, labels): x = features(g) for p in range(4): b[p] += x[p] * y for q in range(4): A[p][q] += x[p] * x[q] return solve(A, b) W = train_model(TRAIN) SCORES = [[sum(a * c for a, c in zip(W, features(g))) for g in frames] for frames, _ in TEST] INDEPENDENT = [[1 if s >= 0.5 else 0 for s in d] for d in SCORES] def measure(decisions): correct = sum(k == y for kd, (_, e) in zip(decisions, TEST) for k, y in zip(kd, e)) flips = sum(sum(kd[t] != kd[t - 1] for t in range(1, FRAMES)) for kd in decisions) return correct / TOTAL, flips d, fl = measure(INDEPENDENT) print(f"independent frame decision: frame accuracy {d:.4f}, instability {fl} transitions " f"({fl / len(TEST):.2f} per sequence)") print(f"difference from the baseline: in accuracy {d - BASELINE:+.4f}, in instability {fl - 0:+d}") ``` ``` independent frame decision: frame accuracy 0.8634, instability 99 transitions (2.75 per sequence) difference from the baseline: in accuracy +0.1620, in instability +99 ``` On the first metric the model passes the baseline by 0.1620, and this says that the per-frame decision really learned something. On the second metric the same model is **99 transitions** above the baseline. The place to read the number is the truth's 71: the model changes its mind **28 more times** than reality changes, 39% more in ratio. Looking per sequence, where there are 1.97 transitions in reality, there are 2.75 transitions in the decision. This excess hardly shows up in frame accuracy. In a sequence, changing its mind on a single frame in the middle of the event takes only 1/432 away from accuracy, but it adds two transitions to instability. In a coverage network the counterpart of this difference is concrete: it means that two separate alerts are produced for the same region in two consecutive readings, and for the party that reads the alert this is more expensive than a loss of accuracy. #### The Decision That Uses Time with a Transition Penalty ```python def penalized_path(score, penalty): """The decision that uses time: frame cost + a fixed penalty per transition.""" cost = [[(score[0] - s) ** 2 for s in (0, 1)]] back = [[0, 0]] for t in range(1, len(score)): new, trace = [0.0, 0.0], [0, 0] for s in (0, 1): best, arg = None, 0 for p in (0, 1): m = cost[t - 1][p] + (0.0 if p == s else penalty) if best is None or m < best: best, arg = m, p new[s], trace[s] = best + (score[t] - s) ** 2, arg cost.append(new) back.append(trace) s = 0 if cost[-1][0] <= cost[-1][1] else 1 path = [s] for t in range(len(score) - 1, 0, -1): s = back[t][s] path.append(s) return path[::-1] print(f"{'penalty':>8}{'frame accuracy':>16}{'instability':>13}{'per sequence':>13}") for penalty in (0.00, 0.05, 0.15, 0.40, 1.00, 4.00): d, fl = measure([penalized_path(score, penalty) for score in SCORES]) print(f"{penalty:>8.2f}{d:>16.4f}{fl:>13}{fl / len(TEST):>13.2f}") print(f"{'truth':>8}{1.0:>16.4f}{TRUE_INSTABILITY:>13}" f"{TRUE_INSTABILITY / len(TEST):>13.2f}") print(f"{'baseline':>8}{BASELINE:>16.4f}{0:>13}{0.0:>13.2f}") TEMPORAL = [penalized_path(score, 0.15) for score in SCORES] worst = max(range(len(TEST)), key=lambda i: sum(INDEPENDENT[i][t] != INDEPENDENT[i][t - 1] for t in range(1, FRAMES))) print(f"test sequence that changed its mind the most (at position {worst + 1}):") for name, d in (("truth", TEST[worst][1]), ("independent", INDEPENDENT[worst]), ("temporal", TEMPORAL[worst])): print(f" {name:<11} " + "".join(str(v) for v in d)) ``` ``` penalty frame accuracy instability per sequence 0.00 0.8634 99 2.75 0.05 0.8657 86 2.39 0.15 0.8611 71 1.97 0.40 0.8472 45 1.25 1.00 0.7801 21 0.58 4.00 0.7014 1 0.03 truth 1.0000 71 1.97 baseline 0.7014 0 0.00 test sequence that changed its mind the most (at position 18): truth 000111110000 independent 000101100010 temporal 000101100000 ``` The first row of the table is the numbers of the second section: zero penalty is the independent frame decision itself. At zero penalty every frame is rounded to the value its own score is closest to, and the sequence plays no role. As the penalty grows, the same scores stay the same — the model, the weights, and the frame scores never change — but the decisions change. So where the difference comes from can be said precisely. The representation did not change: the same three numbers per frame. The model did not change: the same linear function, the same weights. The only thing that changed is **the assumption that the decision rarely changes over time**, and that assumption was entered with a single number, the transition penalty. Six rows are six separate outputs of the same model. The penalty itself is an assumption turned into a number: the cost of a transition is written in the same unit as the cost of being wrong on a frame. A penalty of 0.15 means accepting a transition only if the frame evidence that justifies that transition is strong enough. This is an indirect assumption about **how long the event lasts**, and in the fiction the events last between two and five frames. The trade-off is read in three places. Accuracy rises to its highest value, **0.8657**, at a penalty of 0.05, and instability falls from 99 to 86; a small penalty gains both accuracy and stability. At a penalty of 0.15 instability settles on **71**, the truth's number, and the price of this is giving up 0.0046 from the highest accuracy. When the penalty keeps growing, accuracy falls: 0.8472 at 0.40, 0.7801 at 1.00. The last row pays off the second claim of the course for this lesson. At a penalty of 4.00 the decision sequence falls almost to a constant — **1 transition** over 36 sequences — and the frame accuracy falls to **0.7014**, the baseline's own number. Here the domain baseline is not a separate procedure but **the extreme value of the penalty**; the temporal correction applied with an infinite penalty gives the same output as writing the dominant class into every frame. A setting that tries to reduce instability alone ends up at the procedure that has learned nothing. The last three lines show this in a single sequence. The true sequence carries an event of five frames; the independent decision misses one frame in the middle of the event and gives a false alarm on one frame after the event ends, six transitions. The decision that uses time erases the false alarm and falls to four transitions, but it **cannot fill** the gap in the middle: the score in that frame is low enough, and a penalty of 0.15 is not enough to flip it. A transition penalty smooths the decision; it does not bring back the missed frame. #### Summary - The per-frame decision has two metrics: frame accuracy, and instability, which counts how many times the decision changes its mind across consecutive frames. The baseline gets 0.7014 on the first and 0 on the second. - In instability, small is not good; the number to read is the instability of the true label sequences, and that number is 71 transitions over 36 sequences. - The independent frame decision produces 99 transitions with 0.8634 accuracy: 28 more than the truth, 2.75 per sequence instead of 1.97. This excess is almost invisible in frame accuracy. - The transition penalty produces six separate outputs without changing the representation or the model; at 0.05 accuracy rises to its highest, 0.8657, and at 0.15 instability settles on the truth's 71. - At a penalty of 4.00 the decision falls to the baseline itself (0.7014 and 1 transition): on this metric the domain baseline is not a separate procedure but the extreme value of the penalty. #### Next Step In five lessons the input was always of the same kind: a gray-level grid, as a single frame or as a short sequence. In the first eight lessons of this course the input was a text, and the same fiction came from the notes of the measurement network. The two modalities have so far been measured separately; yet the **note** that the field team wrote for an event and the **coverage frame** of the same event belong to the same event. The next lesson brings the two together in a single space and asks this: how far above random matching does a representation that says whether a note and an image belong to the same event get, and how much of this does a single modality give on its own. ### Multimodal Models Source: https://academia.sh/en/courses/nlp-and-vision/multimodal-models Measuring a joint representation, which brings the note and the coverage frame of the same event together in a single space, against its own baseline: in rooms of twenty-five candidates, random matching gives 0.0400 while the joint representation learned with a contrastive objective gives 0.1660, and the ceiling of the shared information carried by the fiction is 0.2659. On the event kind decision, the most frequent class baseline is 0.4420, note only 0.8940, frame only 0.6140, joint representation 0.8660: the representation that wins at matching stays below what the note alone gives in classification. The course wrap-up collects the domain baseline of fourteen lessons, the method's number, and the difference in a single table. Five lessons measured images and the first eight lessons measured text on its own; yet the **note** the field crew files for an event and the **coverage frame** of the same event belong to the same event. The measure of this lesson is not an accuracy but **matching**: given a note, can we pick the right one among the candidate frames. Its baseline is **random matching**; its ceiling is not 1, it is whatever information the two modalities **share**, and that is computed inside the lesson. - **CV48. The event record is fiction and has two modalities.** The note generator is the same as in the `nlp` topic, the frame is the same as the 12x12 gray-level fiction of lesson 04. Seed 20260218, split 1,100/500. - **CV49. The two modalities do not carry the same information equally.** The frame carries the event kind with a pattern (no block, an unreadable band, one block, two blocks) and the line with a faint band; the note carries the kind with words but writes the line only some of the time. There is noise on both sides: a borrowed side record in the note, and in the frame a missing pattern and an unexpected bright spot. - **CV50. The matching metric is 25 candidates per room.** The ceiling is computed from indistinguishable candidates: two events that carry the same kind and, if the line is written in the note, the same line cannot be told apart. - **CV51. The joint representation is two linear projections**: one from the vocabulary, the other from the frame, into the same space. It is learned with a **contrastive objective**: the similarity of the correct pair is raised, and those of the other twenty-four pairs in the batch are lowered. - **CV52. The event kind decision is read on three representations with the same procedure:** cosine similarity to the class mean, with the most frequent class as the baseline. The vocabulary, the scale, and the weights are learned from training only; no library is called. #### The Two Modalities of the Same Event ```python # event.py — MODEL: Event record fiction: the NOTE of the same event (from the nlp topic) and its # COVERAGE FRAME (the 12x12 gray-level fiction of lesson 04). Seed 20260218. import math SEED, M32, SIZE = 20260218, 0xFFFFFFFF, 12 BACKGROUND, BLOCK, NOISE = 0.30, 0.74, 0.30 KINDS = ("ordinary", "unreachable", "meter fault", "illicit use") SHARE, PATTERN = (0.46, 0.22, 0.19, 0.13), (0, 0, 1, 2) PHRASES = [s.split("|") for s in ( "okuma normal|okuma normaldi|sayac erisimi saglandi|sayaca erisildi|olcum tekrarlandi|" "olcumu yineledik|vana kontrol edildi|vanada sorun yok|kapak temiz|kapakta iz yok|" "gosterge okunakli|gostergede rakam net", "abone evde degildi|abone bulunamadi|erisim saglanamadi|erisim yok|kapi zili calismiyor|" "zil calismadi|bahce kapisi kilitli|bahce kapisi kapali|bodrum katina inilemedi|" "bodrum kilitli|adres bulunamadi|adres eksik", "gosterge donmuyor|gostergede hareket yok|sayac cami kirik|sayacin cami catlak|" "rakam okunmuyor|rakamlar silik|ibre sabit kalmis|ibre kilitli|mekanizma takilmis|" "mekanizmada takilma|sayac durmus|sayac calismiyor", "muhur izi var|muhurde iz goruldu|muhur kopmus|muhurler kopuk|baglanti degistirilmis|" "baglantida degisiklik|kelepce gevsek|kelepcede gevseme|boru ek yeri yeni|boruda yeni ek|" "sayac ters takilmis|sayaci ters takmislar")] EXTRA = ("arac park halinde|kapak kirli|hava yagisli|adres teyit edildi|" "not kisa tutuldu|ikinci ziyaret|bahce icinde|kaldirim uzerinde").split("|") LINES = "kuzey hatti|guney hatti|dogu hatti|bati hatti|merkez hatti".split("|") def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def event(number): """Two modalities of the same event: a short Turkish field note and a 12x12 coverage frame.""" r, k, t = rng(SEED + 41000 + number), 0.0, 0 u = r() for i, p in enumerate(SHARE): k += p if u < k: t = i break parts = [PHRASES[t][int(r() * 12)]] if r() < 0.55: parts.append(PHRASES[t][int(r() * 12)]) if r() < 0.34: # side record borrowed from another kind parts.append(PHRASES[int(r() * 4)][int(r() * 12)]) if r() < 0.62: parts.append(EXTRA[int(r() * 8)]) line, written = int(r() * 5), r() < 0.30 # the line is written in the note only some of the time if written: parts.append(LINES[line]) q = rng(SEED + 600000 + number) g = [[BACKGROUND + (q() + q() + q() - 1.5) * NOISE for _ in range(SIZE)] for _ in range(SIZE)] for i in range(SIZE): # the line that is read appears as a faint band g[i][line * 2] += 0.16 g[i][line * 2 + 1] += 0.16 def block(sy, sx): for i in range(sy, min(SIZE, sy + 2 + int(q() * 2))): for j in range(sx, min(SIZE, sx + 2 + int(q() * 2))): g[i][j] = BLOCK + (q() + q() + q() - 1.5) * NOISE for b in range(PATTERN[t]): if q() >= 0.12: # in some frames the pattern is missing block(int(q() * (SIZE - 3)), line * 2 + b * 2) if t == 1: # not reached: unreadable band for i in range(SIZE): g[i][line * 2] = g[i][line * 2 + 1] = 0.02 + q() * 0.05 if q() < 0.15: # unexpected bright spot elsewhere block(int(q() * (SIZE - 3)), int(q() * (SIZE - 3))) return {"text": ", ".join(parts), "kind": KINDS[t], "line": line, "written": written, "frame": g} DATA = [event(i) for i in range(1600)] TEST = DATA[1100:] count, VOCAB, seen = {}, {}, {} for x in DATA[:1100]: count[x["kind"]] = count.get(x["kind"], 0) + 1 for s in set(x["text"].split()): seen[s] = seen.get(s, 0) + 1 for s in sorted(seen): if seen[s] >= 2: VOCAB[s] = len(VOCAB) MOST_COMMON = max(count, key=count.get) BASELINE = sum(x["kind"] == MOST_COMMON for x in TEST) / len(TEST) print(f"fiction: 1600 events, train 1100, held-out {len(TEST)}; vocabulary {len(VOCAB)} words") print(f"event kind baseline (most frequent class '{MOST_COMMON}') {BASELINE:.4f}; share of events " f"with the line written in the note {sum(x['written'] for x in TEST) / len(TEST):.4f}") x = DATA[1124] print(f"note: {x['text']} -> {x['kind']}, line {x['line']}") for row in x["frame"]: print(" " + "".join(str(min(9, max(0, int(v * 10)))) for v in row)) ``` ``` fiction: 1600 events, train 1100, held-out 500; vocabulary 154 words event kind baseline (most frequent class 'ordinary') 0.4420; share of events with the line written in the note 0.2860 note: boruda yeni ek, dogu hatti -> illicit use, line 2 143245002250 114254245050 423337867534 531264793421 351236684120 231245223341 212444253203 435135111311 154587830332 622058911323 222433403040 241331231422 ``` The event kind baseline is **0.4420** and is the same as the number measured in the `nlp` topic. The second number sets up the problem of the lesson: the line is written in the note for only **0.2860** of the test events, while in the frame it is always visible. In the printed example the note says "dogu hatti" (east line); in the frame that band stands at the fifth and sixth columns, and the two blocks of the suspected illicit use ("illicit use") sit on top of it. #### The Joint Representation and Matching ```python FEATURES, DIM, SCALE, BATCH, CANDIDATES = 15, 12, 10.0, 25, 25 def frame_features(g): """Twelve column means, the mean, the brightest 3x3, the share above the threshold.""" return [sum(g[i][j] for i in range(SIZE)) / SIZE for j in range(SIZE)] + [ sum(sum(s) for s in g) / (SIZE * SIZE), max(sum(g[i + a][j + b] for a in range(3) for b in range(3)) / 9.0 for i in range(SIZE - 2) for j in range(SIZE - 2)), sum(1 for s in g for v in s if v > 0.55) / (SIZE * SIZE)] WORDS = [[VOCAB[s] for s in x["text"].split() if s in VOCAB] for x in DATA] FRAME_FEATURES = [frame_features(x["frame"]) for x in DATA] for k in range(FEATURES): # the scale is learned from the training set only m = sum(o[k] for o in FRAME_FEATURES[:1100]) / 1100 sd = math.sqrt(sum((o[k] - m) ** 2 for o in FRAME_FEATURES[:1100]) / 1100) for o in FRAME_FEATURES: o[k] = (o[k] - m) / sd r = rng(SEED + 77) WN = [[(r() * 2 - 1) * 0.5 for _ in range(DIM)] for _ in range(len(VOCAB))] WF = [[(r() * 2 - 1) * 0.3 for _ in range(DIM)] for _ in range(FEATURES)] def embed(i): """Projections that carry the note and the frame into the same space. MODEL.""" c = [] for v in ([sum(WN[w][d] for w in WORDS[i]) for d in range(DIM)], [sum(FRAME_FEATURES[i][k] * WF[k][d] for k in range(FEATURES)) for d in range(DIM)]): n = math.sqrt(sum(a * a for a in v)) + 1e-9 c.append(([a / n for a in v], n)) return c for step in range(800): # CONTRASTIVE OBJECTIVE start = (step * BATCH) % (1100 - BATCH) E = [embed(i) for i in range(start, start + BATCH)] U, V = [e[0][0] for e in E], [e[1][0] for e in E] dU = [[0.0] * DIM for _ in E] dV = [[0.0] * DIM for _ in E] for i in range(BATCH): # correct pair close, the other 24 pairs far s = [SCALE * sum(a * b for a, b in zip(U[i], V[j])) for j in range(BATCH)] e = [math.exp(v - max(s)) for v in s] t = sum(e) for j in range(BATCH): k = SCALE * (e[j] / t - (1.0 if i == j else 0.0)) for d in range(DIM): dU[i][d] += k * V[j][d] dV[j][d] += k * U[i][d] for p, i in enumerate(range(start, start + BATCH)): for side, (dv, W) in enumerate(((dU[p], WN), (dV[p], WF))): u, n = E[p][side] dot = sum(a * b for a, b in zip(u, dv)) grad = [(dv[d] - u[d] * dot) / n for d in range(DIM)] for w in (WORDS[i] if side == 0 else range(FEATURES)): for d in range(DIM): W[w][d] -= 0.15 * grad[d] * (1.0 if side == 0 else FRAME_FEATURES[i][w]) JOINT = [[embed(i)[y][0] for y in (0, 1)] for i in range(1600)] correct = ceiling = 0 for start in range(1100, 1600, CANDIDATES): room = list(range(start, start + CANDIDATES)) for i in room: correct += i == max(room, key=lambda j: sum(a * b for a, b in zip(JOINT[i][0], JOINT[j][1]))) same = [j for j in room if DATA[j]["kind"] == DATA[i]["kind"]] ceiling += 1.0 / len([j for j in same if DATA[j]["line"] == DATA[i]["line"]] if DATA[i]["written"] else same) print(f"matching, {CANDIDATES} candidates per room, {len(TEST)} held-out events; " f"random matching {1.0 / CANDIDATES:.4f}") print(f"joint representation {correct / len(TEST):.4f}, ceiling of the fiction " f"{ceiling / len(TEST):.4f}") ``` ``` matching, 25 candidates per room, 500 held-out events; random matching 0.0400 joint representation 0.1660, ceiling of the fiction 0.2659 ``` The baseline is **0.0400**: picking one of twenty-five candidates at random. The joint representation gives **0.1660**, more than four times the baseline. The third number says why this metric cannot be read alone: the information the fiction carries in both modalities consists of the kind and, if it is written in the note, the line, and the ceiling computed from those groups is **0.2659**. Compared with 1.0000, 0.1660 looks like a failure; compared with 0.2659, **62% of what can be learned** has been learned. The ceiling of a metric is a number that must be measured as much as its baseline. The source of the difference is the objective: both projections are linear and do nothing on their own. When the contrastive objective pulls the correct pair closer and pushes the wrong ones apart, the projections are forced to turn whatever the two modalities share in the same direction. #### One Modality on Its Own ```python def class_mean(X): """Cosine similarity to the mean vector of each class. MODEL.""" def unit(v): n = math.sqrt(sum(a * a for a in v)) + 1e-9 return [a / n for a in v] M = {} for t in KINDS: d = [i for i in range(1100) if DATA[i]["kind"] == t] M[t] = unit([sum(X[i][k] for i in d) / len(d) for k in range(len(X[0]))]) return sum(max(KINDS, key=lambda t: sum(a * b for a, b in zip(unit(X[i]), M[t]))) == DATA[i]["kind"] for i in range(1100, 1600)) / len(TEST) BAG = [[1.0 if k in WORDS[i] else 0.0 for k in range(len(VOCAB))] for i in range(1600)] print(f"event kind decision, with class mean; baseline {BASELINE:.4f}") for name, X in (("note only", BAG), ("frame only", FRAME_FEATURES), ("joint representation", [JOINT[i][0] + JOINT[i][1] for i in range(1600)])): print(f" {name:<21} {class_mean(X):.4f}") ``` ``` event kind decision, with class mean; baseline 0.4420 note only 0.8940 frame only 0.6140 joint representation 0.8660 ``` The three rows were read with the same procedure and against the same baseline; only the representation changes. Note only **0.8940**, frame only **0.6140**: both are above the baseline, but the frame carries only a part of the event kind, because two of the four kinds carry no block in the frame. The third row says what the joint representation is and what it is not: **0.8660**, only 0.0280 below the note only. This representation was learned **for matching** and beat the baseline fourfold there; on the event kind it cannot reach what the note gives on its own. Bringing two modalities together does not pay off automatically, and the quality of a representation depends on the task in which it is measured. #### Summary - The two modalities do not carry the same information equally: the frame always shows the line, while only 0.2860 of the notes write it. - The joint representation gives 0.1660 at matching; the baseline is 0.0400 and the ceiling of the fiction is 0.2659, that is, 62% of what can be learned. - On the event kind, the baseline is 0.4420, note only 0.8940, frame only 0.6140, joint representation 0.8660: a representation that beats the baseline fourfold on the task it was trained for can fall below a single modality on another task. #### Course Wrap-Up The accounting of fourteen lessons is in a single table, and every number was read from its own lesson. **The rows do not share a common unit**; the last column writes the metric and the direction of each row. **The difference is given with a single sign convention: the method's number minus the domain baseline.** When the direction is small is good, a plus difference says the baseline won. | Lesson | Domain baseline | Method's number | Difference | Metric | |---|---|---|---|---| | Text 01 | 748 | 641 | −107 | vocabulary size, small is good | | Text 02 | 0.7383 | 0.8217 | +0.0833 | accuracy, big is good | | Text 03 | 0.4938 | 0.6049 | +0.1111 | neighborhood hit, big is good | | Text 04 | 0.5061 | 0.9824 | +0.4763 | token accuracy, big is good | | Text 05 | 0.4420 | 0.9620 | +0.5200 | document accuracy, big is good | | Text 06 | 0.4420 | 0.7480 | +0.3060 | document accuracy, big is good | | Text 07 | 0.7777 | 0.9445 | +0.1668 | balanced overlap, big is good | | Text 08 | 0.3970 | 0.8378 | +0.4408 | token accuracy, big is good | | Vision 01 | 0.4875 | 0.9250 | +0.4375 | accuracy, big is good | | Vision 02 | 0.5500 | 0.6687 | +0.1187 | accuracy, big is good | | Vision 03 | 0.3011 | 0.9076 | +0.6065 | mean overlap, big is good | | Vision 04 | 0.8731 | 0.9851 | +0.1120 | pixel accuracy, big is good | | Vision 05 | 0 | 99 | +99 | instability, small is good | | Vision 06 | 0.0400 | 0.1660 | +0.1260 | matching, big is good | **The rule of the course is this: a domain metric cannot be read without writing the number of the dumbest procedure of that domain on the same metric; a number whose domain baseline is not written is an unmeasured number.** The second claim of the course is read from the same table. In seven lessons the domain baseline turned out surprisingly high: 0.8731 in segmentation, 0.7777 in text 07, 0.7383 in text 02, 0.5061 in text 04 — while the same procedure gets 0.0000 at the entity level — and 0.5500 in vision 01 and 02, and in vision 05 an instability of 0, which is an unbeatable number on this metric. In two lessons a procedure that had learned nothing came in **first**: in text 07 the procedure that copies the input, in text 08 the procedure that marks every token, both 1.0000 in the recall direction of the overlap. The metrics that hide this come in four forms: **accuracy read as a share on an imbalanced class**, **accuracy read in a unit other than the one counted**, **overlap read in one direction only**, and **stability that a constant output wins by itself**. In thirty-eight lessons — fourteen of them specific to the domain — the model looked at an input and produced an output, and the correct answer was either at hand or a metric described it. What is learned in a setting where the model learns the **result of the decision only after it is made** and the decision itself **changes the next input** was never asked. The next course, Reinforcement Learning and Recommender Systems, opens with this question.