Text Generation and Summarization

In six lessons the output of the model was always a label and the correct answer was at hand. When a summary of a field note is requested, this setup collapses: there are several correct summaries that carry the same content, and none of them is written in a list of “correct answers”. The summary the inspector wrote is an example, not the single correct one.

In this case the metric turns into the overlap between the candidate text and the reference text: how many words and how many word pairs are shared. The metric thus becomes computable — but what it rewards is a separate question, and this lesson answers that question with a number. The harshest form of the answer is this: a procedure that gives the input back as it is, one that has learned nothing, gets the highest number that can be obtained on this metric.

• NL56. The text fiction is the same as that of the previous lessons and is fiction; seed 20260218.
• NL57. The reference summary is also fiction and is produced by a rule: the phrases that come from the kind of the note itself are kept; the borrowed side record, the noise phrase and the line marker are dropped. This means that the summary is built from the words of the note itself, and it is the source of the result of the lesson.
• NL58. The fiction assumes that the phrase that reports the decision is written at the beginning of the note. The number of the “first phrase” procedure on the content measure is a consequence of this assumption, not a superiority of method.
• NL59. Overlap is computed separately in two directions: the precision side (how many of the candidate’s n-grams exist in the reference) and the recall side (how many of the reference’s n-grams exist in the candidate). The balanced form is the harmonic mean of the two. The metric is measured on single words (1-grams) and word pairs (2-grams).
• NL60. Repeated n-grams are counted by clipping: an n-gram in the candidate is counted as many times as it occurs in the reference. Otherwise a candidate that repeats the same word would inflate precision.
• NL61. The domain baseline is to give the input back as it is. In a summarization task this is the dumbest procedure: it makes no choice, learns nothing, and writes the input as the output.
• NL62. The content check is a number separate from the metric: is the set of event kinds the candidate points to equal to the real kind of the note. This number measures what the overlap metric cannot see.
• NL63. The weights of the selector and generator procedures are learned from the training set only; the held-out set is the last 500 notes.

Reference Text and Candidate Text

The metric has two ends: the reference text written by the inspector and the candidate text produced by the procedure. The fiction builds the reference text by selecting from the phrases of the note itself — as real summarizers often do. The cost of this selection appears at the end of the lesson.

# FICTIONAL field notes — an extended version of the note field of the Data Preparation and Feature Engineering course. Seed 20260218.

SEED, M32 = 20260218, 0xFFFFFFFF
KINDS = ("ordinary", "unreachable", "meter fault", "illicit use")
SHARES = (0.46, 0.22, 0.19, 0.13)
PHRASES = {t: s.split("|") for t, s in zip(KINDS, (
    "okuma normal|okuma normaldi|sayac erisimi saglandi|sayaca erisildi|olcum tekrarlandi|"
    "olcumu yineledik|vana kontrol edildi|vanada sorun yok|kapak temiz|kapakta iz yok|"
    "gosterge okunakli|gostergede rakam net",
    "abone evde degildi|abone bulunamadi|erisim saglanamadi|erisim yok|kapi zili calismiyor|"
    "zil calismadi|bahce kapisi kilitli|bahce kapisi kapali|bodrum katina inilemedi|"
    "bodrum kilitli|adres bulunamadi|adres eksik",
    "gosterge donmuyor|gostergede hareket yok|sayac cami kirik|sayacin cami catlak|"
    "rakam okunmuyor|rakamlar silik|ibre sabit kalmis|ibre kilitli|mekanizma takilmis|"
    "mekanizmada takilma|sayac durmus|sayac calismiyor",
    "muhur izi var|muhurde iz goruldu|muhur kopmus|muhurler kopuk|baglanti degistirilmis|"
    "baglantida degisiklik|kelepce gevsek|kelepcede gevseme|boru ek yeri yeni|boruda yeni ek|"
    "sayac ters takilmis|sayaci ters takmislar"))}
NOISE = ("arac park halinde|kapak kirli|hava yagisli|adres teyit edildi|not kisa tutuldu|"
         "ikinci ziyaret|bahce icinde|kaldirim uzerinde").split("|")
REGIONS = "kuzey hatti|guney hatti|dogu hatti|bati hatti|merkez hatti".split("|")
PHRASE_KIND = {c: t for t in KINDS for c in PHRASES[t]}


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def build_note(n):
    r = rng(SEED + 41000 + n)
    u, k = r(), 0.0
    for i, p in enumerate(SHARES):
        k += p
        if u < k:
            t = KINDS[i]
            break
    p = [PHRASES[t][int(r() * 12)]]
    if r() < 0.55:
        p.append(PHRASES[t][int(r() * 12)])
    if r() < 0.34:
        p.append(PHRASES[KINDS[int(r() * 4)]][int(r() * 12)])
    if r() < 0.62:
        p.append(NOISE[int(r() * 8)])
    if r() < 0.30:
        p.append(REGIONS[int(r() * 5)])
    return {"text": ", ".join(p), "kind": t, "phrases": p,
            "summary": ", ".join(c for c in p if PHRASE_KIND.get(c) == t)}


DATA = [build_note(i) for i in range(1600)]
TRAIN, HELD_OUT = DATA[:1100], DATA[1100:]
for i in (25, 40):
    print(f"note   : {DATA[i]['text']}")
    print(f"summary: {DATA[i]['summary']}  ({DATA[i]['kind']})")
print("avg note", round(sum(len(x["text"].split()) for x in HELD_OUT) / len(HELD_OUT), 2),
      "avg summary", round(sum(len(x["summary"].split()) for x in HELD_OUT) / len(HELD_OUT), 2))

note   : bodrum katina inilemedi, abone evde degildi, sayac cami kirik, bahce icinde
summary: bodrum katina inilemedi, abone evde degildi  (unreachable)
note   : vanada sorun yok, gostergede rakam net, sayac durmus, kuzey hatti
summary: vanada sorun yok, gostergede rakam net  (ordinary)
avg note 6.61 avg summary 3.99

The average note is 6.61 tokens, the average summary 3.99 tokens. The summary is about three fifths of the length of the note and all of it comes from inside the note. These two sentences determine in advance the first row of the next table.

Four Procedures, Six Numbers

Four procedures are compared. Copy the input writes the note as it is. First phrase writes only what comes before the first comma. The learned selector counts, for every phrase, the rate of “how many times did this phrase enter the summary in the training set” and keeps those above one half. The generator procedure copies no phrase: it predicts the event kind from the phrases of the note and writes the canonical sentence of that kind — its output is built from words that do not occur in the note.

def tokenize(m):
    return [w.strip(",").lower() for w in m.split() if w.strip(",")]


def ngrams(z, n):
    return [tuple(z[i:i + n]) for i in range(len(z) - n + 1)]


def overlap(candidate, reference, n=1):
    a, k = ngrams(tokenize(candidate), n), ngrams(tokenize(reference), n)
    if not a or not k:
        return 0.0, 0.0
    match = 0
    remaining = list(k)
    for g in a:
        if g in remaining:
            remaining.remove(g)
            match += 1
    return match / len(a), match / len(k)


# --- procedures
def p_copy(x):
    return x["text"]


def p_first(x):
    return x["phrases"][0]


TALLY = {}
for x in TRAIN:
    for c in x["phrases"]:
        a = TALLY.setdefault(c, [0, 0])
        a[0] += 1
        a[1] += c in x["summary"].split(", ")
SELECT_RATE = {c: (a[1] / a[0]) for c, a in TALLY.items()}


def p_selector(x):
    p = [c for c in x["phrases"] if SELECT_RATE.get(c, 0.0) > 0.5]
    return ", ".join(p) if p else x["phrases"][0]


CANONICAL = {"ordinary": "olagan okuma", "unreachable": "erisim yok",
             "meter fault": "sayac arizasi", "illicit use": "kacak suphesi"}
VOTES = {}
for x in TRAIN:
    for c in x["phrases"]:
        if c in PHRASE_KIND:
            VOTES.setdefault(c, {})
            VOTES[c][x["kind"]] = VOTES[c].get(x["kind"], 0) + 1


def p_generator(x):
    s = {}
    for c in x["phrases"]:
        for t, v in VOTES.get(c, {}).items():
            s[t] = s.get(t, 0) + v
    t = max(s, key=s.get) if s else "ordinary"
    return CANONICAL[t] + " bildirildi"


POINTS_TO = dict(PHRASE_KIND, **{CANONICAL[t]: t for t in KINDS})


def content_ok(candidate, x):
    present = {POINTS_TO[c] for c in POINTS_TO if c in candidate}
    return 1 if present == {x["kind"]} else 0


PROCEDURES = (("copy the input", p_copy), ("first phrase", p_first),
              ("learned selector", p_selector), ("generator procedure", p_generator))
RESULTS = {}
print(f"{'procedure':<19}{'length':>8}{'recall':>9}{'precision':>11}"
      f"{'balanced':>10}{'2-gram balanced':>17}{'content':>9}")
for label, f in PROCEDURES:
    u = r1 = p1 = r2 = p2 = ic = 0.0
    for x in HELD_OUT:
        a = f(x)
        u += len(tokenize(a))
        p, r = overlap(a, x["summary"], 1)
        p1 += p
        r1 += r
        q2, s2 = overlap(a, x["summary"], 2)
        p2 += q2
        r2 += s2
        ic += content_ok(a, x)
    n = len(HELD_OUT)
    u, r1, p1, r2, p2, ic = u / n, r1 / n, p1 / n, r2 / n, p2 / n, ic / n
    g1 = 2 * r1 * p1 / (r1 + p1) if r1 + p1 else 0.0
    g2 = 2 * r2 * p2 / (r2 + p2) if r2 + p2 else 0.0
    RESULTS[label] = (u, r1, p1, g1, g2, ic)
    print(f"{label:<19}{u:>8.2f}{r1:>9.4f}{p1:>11.4f}{g1:>10.4f}{g2:>17.4f}{ic:>9.4f}")
for j, label in ((1, "recall"), (3, "balanced"), (5, "content")):
    s = sorted(RESULTS, key=lambda a: -RESULTS[a][j])
    print(f"{label:<11} ranking: " + " > ".join(s))

procedure            length   recall  precision  balanced  2-gram balanced  content
copy the input         6.61   1.0000     0.6362    0.7777           0.7265   0.7440
first phrase           2.41   0.6901     1.0000    0.8167           0.7576   1.0000
learned selector       4.60   1.0000     0.8948    0.9445           0.9312   0.7440
generator procedure    3.00   0.0622     0.0760    0.0684           0.0114   0.9320
recall      ranking: copy the input > learned selector > first phrase > generator procedure
balanced    ranking: learned selector > first phrase > copy the input > generator procedure
content     ranking: first phrase > generator procedure > copy the input > learned selector

The first row is the debt this lesson pays to the course. The procedure that gives the input back as it is gets 1.0000 on the recall side — the highest number that can be obtained on the scale — and ranks first among the four procedures. This procedure received no training, has no parameters, and makes no choice. The reason for the number lies in the definition of the fiction: because the reference summary is built from the words of the note itself, a candidate that writes the whole note necessarily covers every n-gram of the reference. The recall side measures coverage, and the cheapest way of covering is to write everything.

The same procedure gets 0.6362 on the precision side and 0.7777 in balanced form, and falls to third place. When the three ranking lines are read side by side, the order of the four procedures comes out in three different ways on three metrics; the answer to the question “which procedure is better” depends on which number is printed.

The last row is the second debt. The generator procedure gets 0.9320 on the content check — second among the four procedures — but 0.0684 in the balanced form of the overlap metric and 0.0114 on 2-grams. The metric can hardly see this procedure at all, because it says the right thing with other words. An overlap-based metric measures the sharing of words, not the correctness of the report.

The second row carries a warning. The “first phrase” procedure gets 1.0000 on the content check, and this number is not a superiority of method: the fiction assumes that the phrase that reports the decision is written at the beginning of the note, so taking the first phrase never misses the kind. The same procedure stays at 0.6901 on recall, because in some of the notes the reference summary consists of two phrases and the second is dropped. When it is not written which clause of the fiction a number comes from, this row can be read as “the best summarizer is the shortest one”; when the fiction is written, it cannot be read that way.

The 2-gram column does not repeat the single word column. The balanced form of the copy drops from 0.7777 to 0.7265, that of the selector from 0.9445 to 0.9312, that of the generator procedure from 0.0684 to 0.0114. As the length of the sequence grows, the metric starts to look not only at the choice of words but also at the order, and for a candidate written with other words the penalty grows sixfold.

The distance between the 0.9445 of the learned selector and its 0.7440 on the content measure is the other face of the same gap: the selector finds the words of the reference summary accurately but also keeps the borrowed side record, that is, the event kind the summary reports turns out wrong in one event out of four. The metric does not see this, because that side record is also inside the note, and if it is not in the reference it only lowers precision by a small amount.

What the Metric Rewards

The claim that the recall side rewards coverage can be tested directly: words that do not occur in the reference summary at all are appended to the end of the copy, and one looks at which of the numbers moves.

# What the metric rewards: what happens when unrelated words are appended to the end of the candidate.
FILLER = "kayit alindi ekip donusu form imzalandi saat girildi".split()
print(f"{'candidate':<26}{'length':>8}{'recall':>9}{'precision':>11}"
      f"{'balanced':>10}{'content':>9}")
for e in (0, 2, 4, 8):
    u = r = p = ic = 0.0
    for x in HELD_OUT:
        a = x["text"] + ("" if e == 0 else " " + " ".join(FILLER[:e]))
        u += len(tokenize(a))
        q, s = overlap(a, x["summary"], 1)
        p += q
        r += s
        ic += content_ok(a, x)
    n = len(HELD_OUT)
    u, r, p, ic = u / n, r / n, p / n, ic / n
    print(f"{'copy the input + ' + str(e) + ' filler':<26}{u:>8.2f}{r:>9.4f}"
          f"{p:>11.4f}{2 * r * p / (r + p):>10.4f}{ic:>9.4f}")
r0, p0, g0, i0 = RESULTS["generator procedure"][1], RESULTS["generator procedure"][2], \
    RESULTS["generator procedure"][3], RESULTS["generator procedure"][5]
print(f"{'generator procedure':<26}{RESULTS['generator procedure'][0]:>8.2f}{r0:>9.4f}"
      f"{p0:>11.4f}{g0:>10.4f}{i0:>9.4f}")
below = [c for c in HELD_OUT if overlap(p_copy(c), c["summary"], 1)[1] < 1.0]
print(f"notes where the recall of the copy stays below 1.0000: {len(below)}/{len(HELD_OUT)}")

candidate                   length   recall  precision  balanced  content
copy the input + 0 filler     6.61   1.0000     0.6362    0.7777   0.7440
copy the input + 2 filler     8.61   1.0000     0.4685    0.6381   0.7440
copy the input + 4 filler    10.61   1.0000     0.3750    0.5454   0.7440
copy the input + 8 filler    14.61   1.0000     0.2700    0.4252   0.7440
generator procedure           3.00   0.0622     0.0760    0.0684   0.9320
notes where the recall of the copy stays below 1.0000: 0/500

The recall column is exactly 1.0000 in all four rows. While the length of the candidate text goes from 6.61 to 14.61, more than doubling, this number does not move by a single digit; the content check does not move either, because the appended words point to no event kind. The only number that moves is precision: it drops from 0.6362 to 0.2700 and pulls the balanced form from 0.7777 to 0.4252. In a report that prints only the recall side, appending unrelated words to the candidate text is a free operation.

The last row says that the copy got its 1.0000 on five hundred of five hundred notes. This is not an average but an identity: as long as the reference summary of the fiction is a subset of the note, the copy can never fall below 1.0000 on recall. Most real summaries also carry this property, because summarizers reuse the words of the source. In this setup there is no link between a metric rewarding a procedure and that procedure having learned something.

Summary

• When the output is text there is no single correct answer; the metric turns into the overlap between the candidate text and the reference text and is read separately in two directions.
• The procedure that gives the input back as it is gets 1.0000 on the recall side and is first among the four procedures; it gets 0.6362 on the precision side and falls to third place.
• When eight unrelated words are appended to the end of the copy, recall stays at 1.0000 and precision drops to 0.2700: the recall side of the metric rewards coverage, not correctness.
• The metric sees the sharing of words, not the correctness of the report: the generator procedure gets 0.9320 on the content check, while it gets 0.0684 in balanced form and 0.0114 on 2-grams.
• The same four outputs give three different rankings on three metrics; which procedure is better depends on which number is printed.

Next Step

In this lesson three metrics gave three different rankings and next to each of them was written what was being rewarded. The same problem exists in all the tasks of this course: a number read at the token level and a number read at the entity level rank the same output in two different ways, and on an imbalanced set of classes overall accuracy and the per-class average rank differently. The next lesson is the close of the topic: the same four outputs give four different rankings on four metrics, and which baseline each metric should be read against is written out one by one.