--- title: Distillation source: 'https://academia.sh/en/courses/model-apis/distillation' course: 'Building Applications with Model APIs' language: en updated: '2026-10-21T05:02:32+00:00' license: 'CC BY-SA 4.0' objectives: - 'Compares the small model trained on the output of the large model with the model trained on human labels on the same task.' - 'Computes how much of the teacher accuracy the distilled model captures and how much of the gap to the upper bound it closes.' - 'Shows with numbers why distillation moves backward on a task where the teacher is structurally weak.' - 'Writes the cost of distillation as teacher calls, tokens and the number of human labels.' --- # Distillation Measuring, on two tasks, what a small model trained on 240 fictional records labeled by the large model gains over the same model trained on only six human labels: on the topic task it rises from 0.5000 to 0.6333 and captures the whole teacher accuracy, but it closes only a third of the gap to the 0.9000 given by 240 human labels. On the separate topic-count task distillation moves backward, falling from 0.5833 to 0.3667 and 0.2417, because the teacher does not count. Of the distilled model's errors, 0.9545 are on records where the teacher also errs; for the model trained on six labels this share is 0.3000. In the previous lesson the weights of the large model were in our hands and we moved some of them. In most cases there are no weights in hand; the only thing in hand is the **responses** the large model gives. These responses can be used like a label set and made the input to the training of **a small model**. What is being done is **distillation**: the behavior of the large model is carried over to a model that holds a thousandth as many numbers. The appeal of distillation is plain. A human label is expensive, while a model label is a call. So what has to be measured is not a single number: **how much of the teacher accuracy** the small model captures, where we would have arrived had we spent the same effort on human labels, and **on which task** this transfer does not work at all. - **LA36.** The code is a **simulator**. The large model, the small model and all of the records are **fictional**; no real endpoint is called. Seed **20260218**. - **LA37.** A record is a body sentence and at most two **short suffixes**. One of the labels is the **topic** (the topic of the body, five labels), the other is the **distinct topic count** (the number of distinct topics that appear in the text, four labels). In a share of the records the body does not give the topic away. - **LA38. The large model scores the topic from a key table and does not count the distinct topic count**; it predicts it from the length class. The second behavior is the fictional counterpart of the weakness of language models on counting tasks, and it had been measured in the Introduction to AI Engineering course. - **LA39. The small model** is a single matrix going directly from the token bag to the label; it carries none of the structure the teacher carries. Its training is written inside the lesson. - **LA40.** Three sources are compared: **six human labels**, **two hundred forty teacher labels** and **two hundred forty human labels**. The last is an **upper bound**, not a real option. - **LA41.** The two runs are taken by changing the **labeling seed of the teacher** (20260218 and 20260219); the records and the test set are the same in both runs. ## The Teacher and Two Tasks The limit of distillation is the teacher itself, so the teacher is measured first. The two tasks were chosen to differ on purpose: one is a classification that can be solved by keyword mapping, the other is **counting** the distinct topics that appear in the text. ```python # SIMULATOR -- FICTIONAL. The large model, the small model and all of the records are # written inside the lesson; no real endpoint is called. Seed 20260218. from math import exp SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, a piece is at most five characters b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b TOPICS = ["invoice", "breakdown", "reading", "tariff", "outage"] BODY = [("fatura tutari gecen aya gore yuksek geldi", "invoice"), ("faturami odedim ama sistemde borc gorunuyor", "invoice"), ("evde su akmiyor bir ariza var galiba", "breakdown"), ("baglanti bozuk oldu ve hic calismiyor", "breakdown"), ("sayac okumasi yanlis girilmis endeks tutmuyor", "reading"), ("sayac degisimi icin randevu istiyorum", "reading"), ("hangi tarife basamagindayim ogrenmek istiyorum", "tariff"), ("birim ucret abone tipine gore degisiyor mu", "tariff"), ("mahallede kesinti var ne zaman gelecek", "outage"), ("planli kesinti duyurusu nereden takip edilir", "outage")] SHORT = {"invoice": "fatura notu", "breakdown": "ariza kaydi", "reading": "sayac formu", "tariff": "tarife grubu", "outage": "kesinti plani"} VAGUE = ["gecen hafta bir sorun yasadim yardim gerekiyor", "bu durumu kime iletmem gerektigini bilmiyorum"] def records(n, seed): """A body sentence and at most two short suffixes. The topic is the body's; count is the distinct topic count.""" r, k = rng(seed), [] for i in range(n): c = TOPICS[pick(r(), [0.24, 0.22, 0.18, 0.20, 0.16])] if r() < 0.10: # the body does not give the topic away m, seen = VAGUE[int(r() * 2)], [] else: candidates = [x for x in BODY if x[1] == c] m, seen = candidates[int(r() * len(candidates))][0], [c] for threshold in (0.62, 0.45): if r() < threshold: b = TOPICS[int(r() * 5)] if b not in seen: m += " ayrica " + SHORT[b] seen.append(b) k.append({"text": m, "topic": c, "count": len(seen)}) return k LABELED = records(6, SEED + 1) # a human label is expensive: six records UNLABELED = records(240, SEED + 2) # the unlabeled pile TEST = records(120, SEED + 3) TRACE = {"fatur": "invoice", "tutar": "invoice", "borc": "invoice", "odedi": "invoice", "ariza": "breakdown", "bozuk": "breakdown", "calis": "breakdown", "akmiy": "breakdown", "sayac": "reading", "endek": "reading", "okuma": "reading", "randev": "reading", "tarif": "tariff", "birim": "tariff", "abone": "tariff", "basam": "tariff", "kesin": "outage", "duyur": "outage", "mahal": "outage", "planl": "outage"} TRAP = {"notu": "invoice", "kaydi": "breakdown", "formu": "reading", "grubu": "tariff", "plani": "outage"} # the short suffixes sit in the table at full weight CALLS = [0, 0] def large_model(text, seed, temperature=0.50): """FICTIONAL large model. Scores the topic from the key table; does NOT count for the count task, it predicts from the length class. The second behavior matches the counting weakness of the Introduction to AI Engineering course.""" CALLS[0] += 1 tok = tokenize(text) CALLS[1] += len(tok) p = {k: 0.12 for k in TOPICS} for t in tok: for table, w in ((TRACE, 1.0), (TRAP, 0.70)): if table.get(t) in p: p[table[t]] += w r = rng(seed ^ sum((i + 1) * ord(c) for i, c in enumerate(text))) names = sorted(p) e = [exp(p[a] / temperature) for a in names] s = sum(e) topic = names[pick(r(), [x / s for x in e])] length = 0 if len(tok) <= 7 else (1 if len(tok) <= 10 else 2) count = pick(r(), ([0.10, 0.56, 0.24, 0.10], [0.06, 0.32, 0.40, 0.22], [0.04, 0.20, 0.36, 0.40])[length]) return topic, count def measure(dataset, seed): dt = dc = 0 for x in dataset: k, s = large_model(x["text"], seed) dt += k == x["topic"] dc += s == x["count"] return dt / len(dataset), dc / len(dataset) dist = {} for x in TEST: dist[x["count"]] = dist.get(x["count"], 0) + 1 print("test records", len(TEST), ", distinct topic count distribution", dict(sorted(dist.items()))) print("example record:", TEST[1]["text"]) print(" topic", TEST[1]["topic"], ", distinct topic count", TEST[1]["count"]) print("\nlarge model topic accuracy count accuracy") for name, sd in (("run A (20260218)", SEED), ("run B (20260219)", SEED + 1)): a, b = measure(TEST, sd) print(f"{name:<24} {a:>14.4f} {b:>14.4f}") print(f"cost of the test measurement: calls {CALLS[0]} , input tokens {CALLS[1]}") ``` ``` test records 120 , distinct topic count distribution {0: 4, 1: 38, 2: 63, 3: 15} example record: evde su akmiyor bir ariza var galiba topic breakdown , distinct topic count 1 large model topic accuracy count accuracy run A (20260218) 0.6333 0.3583 run B (20260219) 0.7167 0.3000 cost of the test measurement: calls 240 , input tokens 3214 ``` The large model gives **0.6333 / 0.7167** on the topic task and **0.3583 / 0.3000** on the counting task. The counting task has four labels and the most crowded label holds 63 records of the test set; so a routine that gives every record that label would score 0.5250. The teacher is below it. The difference between the two runs is also 0.0834 on the topic task and 0.0583 on the counting task: the teacher is an uncertain component, and its labels are uncertain too. ## The Share the Small Model Captures The small model is trained on three separate sources in both tasks. The labels of the teacher are produced by making one call for each of the two hundred forty unlabeled records. ```python VOCAB = {} for m in [x[0] for x in BODY] + VAGUE + ["ayrica " + v for v in SHORT.values()]: for t in tokenize(m): VOCAB.setdefault(t, len(VOCAB)) V = len(VOCAB) def split_pieces(text): return sorted({VOCAB[t] for t in tokenize(text) if t in VOCAB}) def target(topic, count, task): return TOPICS.index(topic) if task == "topic" else count def train_model(sample, classes, seed, loops=40, step=0.12): """SMALL model: a single matrix from the token bag directly to the label, V x number of labels.""" r = rng(seed) W = [[(r() - 0.5) * 0.2 for _ in range(classes)] for _ in range(V)] for e in range(loops): for tok, y in sample: o = [sum(W[i][j] for i in tok) for j in range(classes)] mx = max(o) d = [exp(x - mx) for x in o] s = sum(d) d = [x / s for x in d] d[y] -= 1.0 for i in tok: for j in range(classes): W[i][j] -= step * d[j] return W def predict(W, tok, classes): o = [sum(W[i][j] for i in tok) for j in range(classes)] return max(range(classes), key=lambda j: o[j]) def small_accuracy(W, classes, task): return sum(predict(W, split_pieces(x["text"]), classes) == target(x["topic"], x["count"], task) for x in TEST) / len(TEST) CALLS[0] = CALLS[1] = 0 DISTILL = {t: [(x, *large_model(x["text"], t)) for x in UNLABELED] for t in (SEED, SEED + 1)} # the unlabeled pile is being labeled by the teacher LABELING = (CALLS[0], CALLS[1]) SOURCES = [("human labels 6", lambda a, t: [(split_pieces(x["text"]), target(x["topic"], x["count"], a)) for x in LABELED]), ("teacher labels 240", lambda a, t: [(split_pieces(x["text"]), target(k, s, a)) for x, k, s in DISTILL[t]]), ("human labels 240", lambda a, t: [(split_pieces(x["text"]), target(x["topic"], x["count"], a)) for x in UNLABELED])] TEACHER = (measure(TEST, SEED), measure(TEST, SEED + 1)) WEIGHTS_BY = {} print("task small model source accuracy A accuracy B teacher A share captured") for task, classes in (("topic", 5), ("count", 4)): og = TEACHER[0][0 if task == "topic" else 1] for name, build_data in SOURCES: W = train_model(build_data(task, SEED), classes, SEED) a = small_accuracy(W, classes, task) b = small_accuracy(train_model(build_data(task, SEED + 1), classes, SEED + 1), classes, task) WEIGHTS_BY[(task, name)] = (W, classes) print(f"{task:<6} {name:<22} {a:>10.4f} {b:>10.4f} {og:>10.4f} {a / og:>14.4f}") print(f"{task:<6} {'large model (teacher)':<22} {og:>10.4f}" f" {TEACHER[1][0 if task == 'topic' else 1]:>10.4f}") print("\nhow much the small model's error overlaps the teacher's (run A):") for task in ("topic", "count"): for name in ("human labels 6", "teacher labels 240"): W, classes = WEIGHTS_BY[(task, name)] shared = errors = 0 for x in TEST: correct = target(x["topic"], x["count"], task) k, s = large_model(x["text"], SEED) se = predict(W, split_pieces(x["text"]), classes) != correct errors += se shared += se and target(k, s, task) != correct print(f" {task:<6} {name:<22} small errors {errors:>3} , shared {shared:>3}" f" , share {shared / errors:.4f}") print(f"\nsmall model {V} x classes ; human labels 6 or 240 ;" f" teacher calls {LABELING[0] // 2} , input tokens {LABELING[1] // 2}") ``` ``` task small model source accuracy A accuracy B teacher A share captured topic human labels 6 0.5000 0.5583 0.6333 0.7895 topic teacher labels 240 0.6333 0.6917 0.6333 1.0000 topic human labels 240 0.9000 0.9000 0.6333 1.4211 topic large model (teacher) 0.6333 0.7167 count human labels 6 0.5833 0.5833 0.3583 1.6279 count teacher labels 240 0.3667 0.2417 0.3583 1.0233 count human labels 240 0.9417 0.9417 0.3583 2.6279 count large model (teacher) 0.3583 0.3000 how much the small model's error overlaps the teacher's (run A): topic human labels 6 small errors 60 , shared 18 , share 0.3000 topic teacher labels 240 small errors 44 , shared 42 , share 0.9545 count human labels 6 small errors 50 , shared 32 , share 0.6400 count teacher labels 240 small errors 76 , shared 58 , share 0.7632 small model 103 x classes ; human labels 6 or 240 ; teacher calls 240 , input tokens 3256 ``` Distillation works on the topic task. The small model trained on six human labels gives **0.5000 / 0.5583**; the same model trained on two hundred forty teacher labels rises to **0.6333 / 0.6917**. The gain is 0.1333 and the **captured share is 1.0000**: the small model catches up with the accuracy of the teacher exactly. Up to here distillation keeps its promise. The third row shows the limit of the promise. Had the same two hundred forty records been given with **human** labels, the small model would have given **0.9000**. The gap the distilled model closed is only **a third** of the distance between 0.5000 and 0.9000. It is not that the small model cannot carry something; it is the teacher that cannot carry it. Distillation copies not the model but **as much as the teacher saw**. The counting task turns this around and gives the real number of the lesson. The small model trained on six human labels scores **0.5833**. The same model trained on two hundred forty teacher labels **falls** to **0.3667 / 0.2417**. Forty times as many labels, two hundred forty calls and 3256 input tokens were spent, and the result is worse than the model trained on six labels. The teacher does not count on that task, it predicts from the length; distillation faithfully transfers this habit of predicting. The upper bound is **0.9417**: the task is not hard for the small model, it is hard for the teacher. The error overlap column says how faithful the transfer is. On the topic task, **42** of the 44 errors of the distilled model are on records where the teacher also errs; the share is **0.9545**. For the model trained on six human labels the same share is **0.3000**. Both make errors, but the distilled one makes the **same** errors. If two components of a system err in the same place, putting the two side by side catches nothing. Finally, the two run columns show a fragility particular to distillation. When the labeling seed of the teacher changes, the counting accuracy of the distilled model falls **from 0.3667 to 0.2417**; a swing of 0.1250. The rows trained on human labels give the same number in both runs. The uncertainty of the teacher was written into the training set and became permanent there. ## Summary - The large model gives 0.6333 / 0.7167 on the topic task and 0.3583 / 0.3000 on the distinct topic count task; on the counting task it stays below the 0.5250 of a routine that chooses the most crowded label. - On the topic task distillation raises accuracy from 0.5000 to 0.6333 relative to six human labels and captures the whole accuracy of the teacher. - The human labels of the same two hundred forty records give 0.9000; distillation closes only a third of the gap, because the ceiling is the teacher, not the small model. - On the counting task distillation moves backward: from 0.5833 to 0.3667 and 0.2417. The upper bound is 0.9417, so the task is hard not for the small model but for the teacher. - Of the 44 topic errors of the distilled model, 42 are on records where the teacher also errs (share 0.9545); for the model trained on six human labels this share is 0.3000. - The cost is two hundred forty teacher calls and 3256 input tokens; in return, two hundred forty model labels are obtained in place of six human labels, and the run uncertainty of the teacher is inherited. ## Next Step In this lesson the small model was born from the output of the large model. The resulting weights are numbers that no human wrote, derived entirely from the responses of another model. The same holds for the adaptation differences of the previous lesson: in hand is an object **derived** from someone else's model. Running that object on your own server, giving it to subscribers inside a package, selling it to someone else, or serving only its output are not the same thing. The next lesson counts this distinction: when four **license condition classes** are applied to a use scenario, how many distribution options close, which condition alone closes the most options, and how many options remain.