--- title: 'Introduction to Machine Learning' source: 'https://academia.sh/en/courses/introduction-to-machine-learning' language: en updated: '2026-10-07T04:31:19+00:00' license: 'CC BY-SA 4.0' lessons: 11 outcomes: - 'Distinguishes the kinds of learning by problem type and chooses the baseline of each type' - 'Recognizes generalization, overfitting and underfitting by reading the training and held-out set numbers side by side' - 'Describes an end-to-end machine learning workflow and measures the difference inflated by data leakage' --- # Introduction to Machine Learning Showing that the number measured on the data a model has seen and the number measured on the data it has not seen are two separate numbers: the difference between writing rules and learning from examples against the baseline, the number of labels each kind of learning asks for, what a pretext task whose label is derived from the data adds, the four separate baselines of the four problem types, the number moved by going back in the workflow, the same problem turning into a different output in three roles, the reason for separating training, validation and test sets, the point where the two numbers diverge as capacity grows, the decomposition into bias and variance, the two numbers moved by the penalty coefficient, and the difference inflated by leakage. ## Fundamental Concepts Counting what a procedure that learns its decision from data does and how much it gains, together with the baseline: the difference between a hand-written rule set and a learner that derives its rules from data on the same problem, the number of labels asked for by supervised, unsupervised, semi-supervised and reinforcement setups, what a pretext task whose label is derived from the data adds to the actual task, the four separate baselines of classification, regression, clustering and ranking problems, the number of returns in the workflow from problem definition to monitoring, and the same problem turning into a different output in three roles. ### What Is Machine Learning Source: https://academia.sh/en/courses/introduction-to-machine-learning/what-is-machine-learning The same illicit-use suspicion problem is solved with three procedures and three numbers are printed side by side: the non-learning baseline scores 0.7533 on the held-out set, the hand-written rule set written by the field expert scores 0.7507, and the three-question decision tree that derives its rules from data scores 0.8594. The hand-written rule falls 0.0027 below the baseline, the tree rises 0.1061 above it. The difference does not come from the thresholds: when the same thresholds are moved into the hand-written rule, the number drops to 0.7480. Of the 377 subscribers in the held-out set, 93 are suspicious; the hand-written rule flags 151 subscribers and 76 turn out empty, the tree flags 66 subscribers and 13 turn out empty. The dataset is reduced from a readings table of 3,199 rows to a subscriber table of 1,260 rows, seed 20260218. Over ninety lessons, the Data Analytics curriculum measured every step of the path from data to decision and showed at each step that the decision moves the number. One question stayed open at the close: every one of those decisions was made by a **human**. Which rows to count, which threshold to choose, which channel to use. Which number a procedure that **learns its decision from data** moves, and where that procedure's own decisions stop, was never asked. This lesson asks that question and establishes the unit of measurement for the curriculum. From here on, no number a model produces is written alone; a **baseline** stands beside it — the number, on the same measure, of a procedure that tries to solve the same problem without learning. The curriculum's rule follows from this: **a model is as good as the margin by which it beats a procedure that is dumber than itself. A performance with no baseline written next to it counts as unmeasured.** - **FC1.** The dataset is **fictional** and carried over from the Data Analytics curriculum: a readings table in long format with 3,199 rows, 1,329 subscribers, three periods, five regions, seed 20260218. **The label is added in this curriculum and it is fictional too**; the rule that produces the label is written openly in the code and is never given to any model. - **FC2. Machine learning** is deriving the rule from examples instead of writing it by hand. The input is not a rule but a set of examples made of input–output pairs; the output is a decision, and the structure that makes that decision is called a **model**. - **FC3. A feature** is a column given to the model, and **a label** is the column the model is asked to predict. Which column is a feature and which is the label in the same table comes not from the data but from the question being asked. - **FC4. A baseline** is a non-learning procedure and it is **chosen explicitly in every lesson**. In this lesson it is predicting the most frequent class; because the classes are imbalanced, this procedure produces a high accuracy even without learning, and it honestly raises the bar the model has to clear. - **FC5.** Every number is measured on two sets: the **training set** on which the procedure is built and the **held-out set** that the procedure has never seen. Why the split is made this way is not the subject of this topic; the Generalization topic covers it. Here the split is applied as the course's rule. - **FC6.** The measure itself is a decision. Accuracy counts a visit made for nothing and a missed illicit case with the same weight; that weight is a choice and it is written down where it is made. - **FC7.** In this curriculum the models are written **with the standard library and at small scale**. The aim is not to imitate a library but to make the behavior of the procedure measurable line by line. #### The Problem and the Fictional Dataset The question is this: from meter readings, to tell apart the subscriber carrying **illicit-use or fault suspicion** before the field team goes to the door. What exists is the readings table of the measurement network; what is wanted is a decision for each subscriber. The table is first regenerated with the same seed. ```python # data.py — MODEL: the regional measurement network is regenerated with the same seed. # The dataset is fictional; no real subscriber, meter or region is represented. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF PERIODS = ["2026-01", "2026-02", "2026-03"] REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t: int): """Linear congruential generator: the seed is written in the code, the output is the same from run to run.""" x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value() -> float: s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u: float, w: list[float]) -> int: t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 _subscribers, _dropped = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] k = {"subscriber_id": 10001 + i, "region": b[0], "base": b[2], "household_size": pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]), "satisfaction": pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) + 1} (_dropped if r() < 0.046 else _subscribers).append(k) _valid = {a["subscriber_id"] for a in _subscribers} READINGS = [] for k in _subscribers + _dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor(k["base"] * math.exp( (r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] in _valid: READINGS.append({"subscriber_id": k["subscriber_id"], "region": k["region"], "period": PERIODS[d], "consumption_m3": t}) r() print(f"fictional readings table (seed {SEED}): {len(READINGS)} rows, {len(_valid)} subscribers") ``` ``` fictional readings table (seed 20260218): 3199 rows, 1329 subscribers ``` This table cannot be given to a model directly. The decision is per subscriber, whereas the table is per reading, and a subscriber is represented by one, two or three rows. The shape the model will see must be **a single row per subscriber**; each row carries nine numbers that summarize a subscriber and one label. ```python # summary.py — the readings table is reduced to a single row per subscriber and the label is added. # The rule that produces the label is FICTIONAL, written openly in the code and not given to the model. FEATURES = ["mean", "per_person", "drop", "zero_readings", "reading_count", "satisfaction", "household_size", "highest", "lowest"] _groups = {} for o in READINGS: _groups.setdefault(o["subscriber_id"], []).append(o) SUBSCRIBERS = [] for k in _subscribers: o = _groups.get(k["subscriber_id"]) if not o: continue t = [x["consumption_m3"] for x in o] mean = sum(t) / len(t) a = {"subscriber_id": k["subscriber_id"], "region": k["region"], "household_size": k["household_size"], "satisfaction": k["satisfaction"], "reading_count": len(t), "mean": mean, "highest": max(t), "lowest": min(t), "zero_readings": sum(1 for v in t if v == 0), "per_person": mean / max(1, k["household_size"]), "drop": (t[0] - t[-1]) / t[0] if len(t) > 1 and t[0] > 0 else 0.0} r = rng(SEED + 31000 + a["subscriber_id"]) latent = (1.30 * a["zero_readings"] + 1.25 * (a["drop"] > 0.45) + 1.05 * (a["per_person"] > 12.0) + 0.65 * (a["reading_count"] < 3) + 0.45 * (a["satisfaction"] <= 2) - 1.55) a["suspicion"] = 1 if latent + (r() + r() - 1.0) * 1.1 > 0 else 0 SUBSCRIBERS.append(a) r = rng(SEED + 90001) TRAIN, HELD_OUT = [], [] for a in SUBSCRIBERS: (TRAIN if r() < 0.70 else HELD_OUT).append(a) def accuracy(predict, split: list[dict]) -> float: return sum(1 for a in split if predict(a) == a["suspicion"]) / len(split) def positives(k: list[dict]) -> int: return sum(a["suspicion"] for a in k) print(f"model table: {len(SUBSCRIBERS)} subscribers, {len(FEATURES)} features, 1 label") print(f"label: {positives(SUBSCRIBERS)} suspicious, {len(SUBSCRIBERS) - positives(SUBSCRIBERS)} clean, " f"suspicious rate {100 * positives(SUBSCRIBERS) / len(SUBSCRIBERS):.1f} percent") print(f"train {len(TRAIN)} subscribers ({positives(TRAIN)} suspicious), " f"held-out {len(HELD_OUT)} subscribers ({positives(HELD_OUT)} suspicious)") ``` ``` model table: 1260 subscribers, 9 features, 1 label label: 327 suspicious, 933 clean, suspicious rate 26.0 percent train 883 subscribers (234 suspicious), held-out 377 subscribers (93 suspicious) ``` Of the 1,329 subscribers, 1,260 entered the model table; 69 subscribers have no readings at all and there is nothing to summarize for them. The suspicious rate is 26.0 percent. This rate directly sets the bar the model has to clear: a procedure that says "clean" to everyone without thinking is right for 74 percent of the subscribers. The rule that produces the label sits in the block and is not a real illicit-use detection. Zero readings, a sharp drop between periods, high consumption per person, a missing period and low satisfaction are gathered into one score, noise is added on top, and if the score passes zero the subscriber counts as suspicious. The noise is deliberate: a noiseless label turns into a puzzle that any procedure which splits the features finely enough hits exactly, and nothing is measured. #### The Baseline and the Hand-Written Rule Two procedures are written. The first learns nothing: it looks at the most frequent class in the training set and says that to everyone. The second is the rule in the field expert's head — if there is a zero reading, if consumption dropped sharply, or if consumption per person is high, the subscriber is flagged. The thresholds come from experience. ```python # rule.py — two procedures: the non-learning baseline and the hand-written rule set. from collections import Counter most_frequent = Counter(a["suspicion"] for a in TRAIN).most_common(1)[0][0] def baseline(a: dict) -> int: return most_frequent def hand_written(a: dict) -> int: """The rule set the field expert wrote: the thresholds come from experience.""" if a["zero_readings"] >= 1: return 1 if a["drop"] > 0.50: return 1 if a["per_person"] > 15.0: return 1 return 0 W = [31, 12, 12] fmt = lambda h: str(h[0]).ljust(W[0]) + str(h[1]).rjust(W[1]) + str(h[2]).rjust(W[2]) print(fmt(["procedure", "train", "held-out"])) for name, f in [("baseline: most frequent class", baseline), ("hand-written rule set", hand_written)]: print(fmt([name, f"{accuracy(f, TRAIN):.4f}", f"{accuracy(f, HELD_OUT):.4f}"])) ``` ``` procedure train held-out baseline: most frequent class 0.7350 0.7533 hand-written rule set 0.7622 0.7507 ``` The expert's rule beats the baseline by 0.0272 on the training set, but on the held-out set it falls 0.0027 **below** it. Three reasonable rules, three reasonable thresholds and domain knowledge were not enough to beat a procedure that learns nothing. This result is a finding in itself: had the baseline not been written, the 0.7507 in the table could have been reported as "the rule set makes the right decision for three in four subscribers." #### The Learner That Derives Its Rules from Data The third procedure takes the **form** of the rule from the human and the **content** from the data. The form is this: pick a feature, pick a threshold, split the set in two, and repeat the same step in each part. Which feature to pick with which threshold is found by search; the quantity searched for is **how mixed the two parts are in terms of the label** after the split. This structure is called a **decision tree**. ```python # tree.py — the learner that derives its rules from data: a three-question decision tree # built by greedy splitting. No library is called; the split criterion and stopping rule are here. DEPTH, MIN_SIZE = 3, 40 def impurity(k: list[dict]) -> float: """How mixed the two labels are in a set: zero if it has a single label.""" if not k: return 0.0 p = positives(k) / len(k) return 2 * p * (1 - p) def best_split(k: list[dict]): best, choice = impurity(k) * len(k), None for feat in FEATURES: for threshold in sorted({a[feat] for a in k})[1:]: left = [a for a in k if a[feat] < threshold] right = [a for a in k if a[feat] >= threshold] m = impurity(left) * len(left) + impurity(right) * len(right) if m < best - 1e-9: # floating-point equality is not tested, a small margin is left best, choice = m, (feat, threshold, left, right) return choice def majority(k: list[dict]) -> int: return 1 if 2 * positives(k) > len(k) else 0 def fit(k: list[dict], depth: int): b = best_split(k) if depth and len(k) >= MIN_SIZE else None if b is None: return majority(k) left, right = fit(b[2], depth - 1), fit(b[3], depth - 1) return left if left == right else (b[0], b[1], left, right) def apply(node, a: dict) -> int: while isinstance(node, tuple): node = node[2] if a[node[0]] < node[1] else node[3] return node def show(d, indent: str = "") -> None: if not isinstance(d, tuple): print(f"{indent}-> {'suspicious' if d else 'clean'}") return print(f"{indent}{d[0]} < {d[1]:.3f}") show(d[2], indent + " ") show(d[3], indent + " ") TREE = fit(TRAIN, DEPTH) show(TREE) print() print(fmt(["procedure", "train", "held-out"])) FIRST_THREE = [("baseline: most frequent class", baseline), ("hand-written rule set", hand_written), ("tree learned from data", lambda a: apply(TREE, a))] for name, f in FIRST_THREE: print(fmt([name, f"{accuracy(f, TRAIN):.4f}", f"{accuracy(f, HELD_OUT):.4f}"])) ``` ``` drop < 0.461 per_person < 12.060 -> clean reading_count < 3.000 -> suspicious -> clean lowest < 6.000 -> suspicious per_person < 13.052 -> clean -> suspicious procedure train held-out baseline: most frequent class 0.7350 0.7533 hand-written rule set 0.7622 0.7507 tree learned from data 0.8709 0.8594 ``` The tree beats the baseline by 0.1061 on the held-out set. This is the number the lesson measures, and it carries meaning not alone but inside the three-row table: on the same problem the non-learning procedure gives 0.7533, the rule written with domain knowledge 0.7507, and the rule derived from data 0.8594. The thresholds the tree found are close to the thresholds of the rule that produces the label. The label rule uses 0.45 for the drop and 12.0 for consumption per person; the tree finds 0.461 and 12.060. The tree never saw these numbers; it looked only at the features and labels of 883 subscribers. #### Where the Difference Comes From The difference looks as if it comes from the thresholds — the expert guessed 0.50 and 15.0, the tree found 0.461 and 12.060. This explanation can be tested: the thresholds in the expert's rule are replaced with the tree's, and the number is measured again. ```python # separate.py — does the difference come from the thresholds: the expert's rule keeps the same form, # its thresholds are replaced with the values the tree found. def hand_written_tree_thresholds(a: dict) -> int: return 1 if (a["zero_readings"] >= 1 or a["drop"] > 0.461 or a["per_person"] > 12.060) else 0 W2 = [31, 13, 8, 14, 8] fmt2 = lambda h: str(h[0]).ljust(W2[0]) + "".join(str(v).rjust(W2[i + 1]) for i, v in enumerate(h[1:])) print(f"held-out set: {len(HELD_OUT)} subscribers, {positives(HELD_OUT)} suspicious") print(fmt2(["procedure", "flagged", "correct", "empty visit", "missed"])) PROCEDURES = FIRST_THREE + [("hand-written, tree thresholds", hand_written_tree_thresholds)] for name, f in PROCEDURES: flagged = [a for a in HELD_OUT if f(a) == 1] d = positives(flagged) print(fmt2([name, len(flagged), d, len(flagged) - d, positives(HELD_OUT) - d])) print() for name, f in PROCEDURES: print(f"{name:<31}held-out accuracy {accuracy(f, HELD_OUT):.4f}") ``` ``` held-out set: 377 subscribers, 93 suspicious procedure flagged correct empty visit missed baseline: most frequent class 0 0 0 93 hand-written rule set 151 75 76 18 tree learned from data 66 53 13 40 hand-written, tree thresholds 184 91 93 2 baseline: most frequent class held-out accuracy 0.7533 hand-written rule set held-out accuracy 0.7507 tree learned from data held-out accuracy 0.8594 hand-written, tree thresholds held-out accuracy 0.7480 ``` The explanation collapsed. When the expert's rule is run with the tree's thresholds, the number **dropped** from 0.7507 to 0.7480. The difference is not in the thresholds but in **the way the conditions are combined**. The expert's rule joins three conditions with "or": every subscriber for whom any one holds is flagged. The tree uses the same conditions nested — it looks at consumption per person only in the branch where the drop is small, asks the number of readings as a second condition in that branch, and in the branch where the drop is large it moves to an entirely different feature. The count table gives the result. The expert's rule flags 151 subscribers and finds 75 of them correctly; 76 visits go to waste. The tree flags 66 subscribers, finds 53 of them correctly, and 13 visits go to waste. The tree catches fewer illicit cases — 53 against 75 — but it produces about six times fewer empty visits, and because the accuracy measure counts an empty visit with the same weight as a missed illicit case, the tree wins. This is the concrete counterpart of FC6. If the cost of an empty visit is lower than the cost of a missed illicit case, the measure changes, and when the measure changes the ranking can change too: the procedure in the last row catches 91 of the 93 suspicious subscribers and misses only 2, but with 93 empty visits. Which procedure is "good" is determined together with the weighting buried inside accuracy. #### Summary - Machine learning is deriving the rule from examples instead of writing it by hand; in this lesson the rule is a three-question decision tree derived from a training set of 883 subscribers. - In this lesson the baseline is predicting the most frequent class and it scores 0.7533 on the held-out set; because the suspicious rate is 26.0 percent, even a non-learning procedure produces a number that looks high. - The hand-written rule set scores 0.7507 on the held-out set and stays below the baseline; the tree learned from data scores 0.8594 and beats the baseline by 0.1061. - The difference does not come from the thresholds: when the same thresholds are moved into the hand-written rule, the number drops to 0.7480. The difference comes from whether the conditions are nested or joined with "or". - The accuracy measure counts an empty visit and a missed illicit case equally; while the tree catches 53 illicit cases and produces 13 empty visits, the expert's rule catches 75 and produces 76 empty visits, and if the measure changes the ranking can change too. #### Next Step This lesson used a single setup: every subscriber's label was known and the procedure saw all 883 labels. In this curriculum the label was produced for free, not in the field. In reality a label means a field visit, and visits are countable. The next lesson builds the same problem in four separate setups — supervised, unsupervised, semi-supervised and reinforcement — and counts how many labels each setup asks for: how much of the difference above the baseline remains when the work is done with 60 labels instead of all 883, what an unlabeled setup can extract from the same set, and how a procedure that learns the label by visiting spends its budget. ### Types of Learning Source: https://academia.sh/en/courses/introduction-to-machine-learning/types-of-learning The same illicit-use suspicion problem is set up in four setups and the number of labels each setup asks for is counted: supervised 0.8594 on the held-out set with 883 labels, unsupervised 0.8117 with 6 labels, semi-supervised 0.8090 with 60 labels, reinforcement 0.8090 with 120 visits. The baseline is 0.7533. As the label budget falls from 883 to 30, the held-out figure drops from 0.8594 to 0.7745, while the training figure rises from 0.8709 to 0.9400. The two-cluster split sits exactly on the baseline, the six-cluster split gains 0.0584. The visit policy finds 73 illicit cases in 120 visits; if the same budget were spent at random, 25 would be found. The previous lesson used a single setup: the labels of 883 subscribers were known and the procedure saw all of them. The labels were produced for free there. In the field a label means a door visit, and visits are countable. This is what defines a setup — which piece of information is obtained, in what amount, and at what price. This lesson sets up the same problem in four setups and counts **how many labels each setup asks for**. The measure and the held-out set stay the same in all four so that the numbers can be read side by side. - **FC8. Supervised learning**: the label of every example is known. The procedure searches directly for the mapping from feature to label. - **FC9. Unsupervised learning**: there is no label. The output is not a decision but a **structure**; turning the structure into a decision is a separate job and it asks for labels. - **FC10. Semi-supervised learning**: a small number of labeled examples is used together with a large number of unlabeled ones. The contribution of the unlabeled rows is a measurable number. - **FC11. Reinforcement learning**: the label is not given in advance, it comes **as the return on an action**. The procedure chooses whom to visit, sees the label only of the one it visited, and makes its next choice accordingly. - **FC12.** The label is a budget. In this lesson the unit of comparison between the setups is not accuracy but **labels spent per unit of accuracy**. - **FC13.** The real measure of the reinforcement setup is not accuracy but the **return**: the number of illicit cases found with the budget spent. Accuracy is still printed, but the number that describes the setup is the return. #### Rebuilding the Fictional Dataset ```python # setup.py — the fictional dataset of lesson 01 is rebuilt with the same seed: the readings # table, the subscriber summary, the fictional label, the train / held-out split and the tree learner. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] FEATURES = ["mean", "per_person", "drop", "zero_readings", "reading_count", "satisfaction", "household_size", "highest", "lowest"] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 _subscribers, _dropped, _series = [], [], {} for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] k = {"subscriber_id": 10001 + i, "region": b[0], "base": b[2], "household_size": pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]), "satisfaction": pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) + 1} (_dropped if r() < 0.046 else _subscribers).append(k) _valid = {a["subscriber_id"] for a in _subscribers} for k in _subscribers + _dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor(k["base"] * math.exp( (r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] in _valid: _series.setdefault(k["subscriber_id"], []).append(t) r() SUBSCRIBERS = [] for k in _subscribers: t = _series.get(k["subscriber_id"]) if not t: continue mean = sum(t) / len(t) a = {"subscriber_id": k["subscriber_id"], "region": k["region"], "series": t, "household_size": k["household_size"], "satisfaction": k["satisfaction"], "reading_count": len(t), "mean": mean, "highest": max(t), "lowest": min(t), "zero_readings": sum(1 for v in t if v == 0), "per_person": mean / max(1, k["household_size"]), "drop": (t[0] - t[-1]) / t[0] if len(t) > 1 and t[0] > 0 else 0.0} r = rng(SEED + 31000 + a["subscriber_id"]) a["suspicion"] = 1 if (1.30 * a["zero_readings"] + 1.25 * (a["drop"] > 0.45) + 1.05 * (a["per_person"] > 12.0) + 0.65 * (a["reading_count"] < 3) + 0.45 * (a["satisfaction"] <= 2) - 1.55 + (r() + r() - 1.0) * 1.1) > 0 else 0 SUBSCRIBERS.append(a) r = rng(SEED + 90001) TRAIN, HELD_OUT = [], [] for a in SUBSCRIBERS: (TRAIN if r() < 0.70 else HELD_OUT).append(a) positives = lambda k: sum(a["suspicion"] for a in k) accuracy = lambda f, k: sum(1 for a in k if f(a) == a["suspicion"]) / len(k) _impurity = lambda k: 2 * (positives(k) / len(k)) * (1 - positives(k) / len(k)) if k else 0.0 def fit(k, depth=3, min_size=40): """The greedy decision tree of lesson 01: the split criterion and stopping rule are there.""" best, choice = _impurity(k) * len(k), None if depth and len(k) >= min_size: for feat in FEATURES: for threshold in sorted({a[feat] for a in k})[1:]: left = [a for a in k if a[feat] < threshold] right = [a for a in k if a[feat] >= threshold] m = _impurity(left) * len(left) + _impurity(right) * len(right) if m < best - 1e-9: best, choice = m, (feat, threshold, left, right) if choice is None: return 1 if 2 * positives(k) > len(k) else 0 lt, rt = fit(choice[2], depth - 1, min_size), fit(choice[3], depth - 1, min_size) return lt if lt == rt else (choice[0], choice[1], lt, rt) def apply(d, a): while isinstance(d, tuple): d = d[2] if a[d[0]] < d[1] else d[3] return d print(f"fictional dataset (seed {SEED}): {len(SUBSCRIBERS)} subscribers, {len(TRAIN)} train, " f"{len(HELD_OUT)} held-out, {positives(SUBSCRIBERS)} suspicious") ``` ``` fictional dataset (seed 20260218): 1260 subscribers, 883 train, 377 held-out, 327 suspicious ``` #### Supervised: As the Label Budget Shrinks In the supervised setup the only thing that changes is how many labels are given. With small budgets a single draw is misleading, so five separate subsets are drawn for each budget and the numbers are averaged. ```python # supervised.py — the number of labels is a budget: the same procedure is built with more labels. # Five separate random subsets are drawn for each budget and the two numbers are averaged. import statistics from collections import Counter BUDGET, DRAWS = [30, 60, 150, 400, 883], 5 most_frequent = Counter(a["suspicion"] for a in TRAIN).most_common(1)[0][0] BASE_TRAIN = accuracy(lambda a: most_frequent, TRAIN) BASE_HELD = accuracy(lambda a: most_frequent, HELD_OUT) W = [20, 10, 12, 12] fmt = lambda h: str(h[0]).ljust(W[0]) + "".join(str(v).rjust(W[i + 1]) for i, v in enumerate(h[1:])) print(fmt(["labels", "train", "held-out", "vs baseline"])) print(fmt(["baseline (0 labels)", f"{BASE_TRAIN:.4f}", f"{BASE_HELD:.4f}", "0.0000"])) for n in BUDGET: inside, outside = [], [] for j in range(DRAWS): r = rng(SEED + 4100 + j) subset = sorted(TRAIN, key=lambda a: r())[:n] d = fit(subset, 3, max(12, n // 20)) inside.append(accuracy(lambda a: apply(d, a), subset)) outside.append(accuracy(lambda a: apply(d, a), HELD_OUT)) o = statistics.fmean(outside) print(fmt([n, f"{statistics.fmean(inside):.4f}", f"{o:.4f}", f"{o - BASE_HELD:+.4f}"])) ``` ``` labels train held-out vs baseline baseline (0 labels) 0.7350 0.7533 0.0000 30 0.9400 0.7745 +0.0212 60 0.9267 0.7841 +0.0308 150 0.8947 0.7873 +0.0340 400 0.8725 0.8419 +0.0886 883 0.8709 0.8594 +0.1061 ``` The difference over the baseline erodes together with the budget: 0.1061 at 883 labels, 0.0886 at 400, 0.0212 at 30. A model built with thirty labels does beat the baseline, but the margin by which it beats it is about one fifth of that of the model with 883 labels. The two columns move in opposite directions. As labels decrease the held-out figure falls while the **training figure rises**: 0.9400 at 30 labels, 0.8709 at 883 labels. In a set of thirty rows the tree can classify almost every row correctly; on the held-out set the same tree stays at 0.7745. This opposite movement is the subject of the Generalization topic; here it is only put on record. #### Unsupervised: Unlabeled Structure and the Cost of Naming In the unsupervised setup the label column is never read. The procedure looks only at the feature space and divides subscribers who are close to one another into clusters. Because the scales differ, each feature has its mean subtracted and is divided by its own variance; otherwise a column in cubic meters crushes the small-scale columns. The output of this setup is not a decision. The sentence "this subscriber is in the fourth cluster" tells the field team nothing; the **name** of the cluster is needed. Naming asks for labels: one subscriber is visited from each cluster, the name of the cluster becomes the label of that subscriber, and the cost is K labels. ```python # unsupervised.py — no label is used and the cluster structure is derived. The output is not a # decision but a structure; to turn it into a decision one subscriber is visited from each # cluster and the name of the cluster becomes the label of that subscriber: cost K labels. MEANS = {o: statistics.fmean(a[o] for a in TRAIN) for o in FEATURES} STDS = {o: statistics.pstdev(a[o] for a in TRAIN) or 1.0 for o in FEATURES} r = rng(SEED + 4100) MIXED = sorted(TRAIN, key=lambda a: r()) vec = lambda a: [(a[o] - MEANS[o]) / STDS[o] for o in FEATURES] dist = lambda v, m: sum((x - y) ** 2 for x, y in zip(v, m)) def cluster(K): """K centers are picked at random and pulled to the mean of their members over twenty rounds.""" r = rng(SEED + 4202) centers = [vec(MIXED[int(r() * len(MIXED))]) for _ in range(K)] for _ in range(20): groups = [[] for _ in range(K)] for a in TRAIN: v = vec(a) groups[min(range(K), key=lambda i: dist(v, centers[i]))].append(v) for i in range(K): if groups[i]: centers[i] = [sum(c) / len(c) for c in zip(*groups[i])] return lambda a: min(range(K), key=lambda i: dist(vec(a), centers[i])) print(fmt(["clusters", "labels", "train", "held-out"])) CLUSTERS = {} for K in (2, 6): which, names = cluster(K), {} for a in MIXED: names.setdefault(which(a), a["suspicion"]) f = lambda a, h=which, i=names: i[h(a)] CLUSTERS[K] = (which, f, names) print(fmt([K, K, f"{accuracy(f, TRAIN):.4f}", f"{accuracy(f, HELD_OUT):.4f}"])) which6 = CLUSTERS[6][0] print("K=6 cluster sizes", sorted(Counter(which6(a) for a in TRAIN).values(), reverse=True), "size of the cluster named suspicious", sum(1 for a in TRAIN if CLUSTERS[6][2][which6(a)] == 1)) ``` ``` clusters labels train held-out 2 2 0.7350 0.7533 6 6 0.7780 0.8117 K=6 cluster sizes [268, 157, 147, 131, 102, 78] size of the cluster named suspicious 78 ``` The two-cluster split sits **exactly on** the baseline: 0.7533, a difference of zero. Both clusters were named clean, which means the procedure made the same decision as the baseline that says "clean" to everyone. Clustering does not split by suspicion; it splits by the largest variance, and in this set the largest variance lies in the size of consumption, not in illicit use. With the six-cluster split the picture changes: 0.8117, 0.0584 above the baseline. The smallest cluster, with 78 subscribers, was named suspicious, and one of the six labels was enough to establish this. The number of clusters is a decision and its effect is 0.0584 — a difference produced not by the clustering but by **how many clusters** were asked for. #### Semi-Supervised: The Contribution of Unlabeled Rows In the semi-supervised setup the labels of 60 subscribers are known and those of 823 are not. The unlabeled rows are not discarded: according to the cluster structure each one is given a **pseudo-label**, and the tree is built on the sum of 60 true labels and 823 pseudo-labels. ```python # semi.py — 60 labels + 823 unlabeled rows: the unlabeled rows get a pseudo-label from the # cluster structure and the tree is built with 60 true labels and 823 pseudo-labels. LABELED, UNLABELED = MIXED[:60], MIXED[60:] name60 = {} for i in range(6): group = [x["suspicion"] for x in LABELED if which6(x) == i] name60[i] = 1 if group and 2 * sum(group) > len(group) else 0 PSEUDO = [{**a, "suspicion": name60[which6(a)]} for a in UNLABELED] tree60, tree_semi = fit(LABELED, 3, 12), fit(LABELED + PSEUDO, 3, 40) f60, f_semi = lambda a: apply(tree60, a), lambda a: apply(tree_semi, a) agree = sum(1 for p, a in zip(PSEUDO, UNLABELED) if p["suspicion"] == a["suspicion"]) print(f"pseudo-labels {len(PSEUDO)}, matching the true label {agree}, " f"rate {agree / len(PSEUDO):.4f}") print(fmt(["setup", "labels", "train", "held-out"])) for setup, f in [("60 labels only", f60), ("60 + 823 unlabeled", f_semi)]: print(fmt([setup, 60, f"{accuracy(f, TRAIN):.4f}", f"{accuracy(f, HELD_OUT):.4f}"])) ``` ``` pseudo-labels 823, matching the true label 633, rate 0.7691 setup labels train held-out 60 labels only 60 0.7792 0.7905 60 + 823 unlabeled 60 0.7780 0.8090 ``` The same 60 labels, two setups, two numbers. When the 823 unlabeled rows are added, the held-out figure goes from 0.7905 to 0.8090: **0.0185**. Only 76.91 percent of the pseudo-labels match the true label, which means nearly a quarter of them are wrong, and the contribution is still positive. What the unlabeled rows carry is not a label but **density information**: which combinations of features are frequent and which are rare. #### Reinforcement: Earning the Label with an Action In the fourth setup the label column is closed. The procedure picks a subscriber, that subscriber is visited, and only that subscriber's label is opened. The budget is six rounds of twenty visits each. The first round is random; in the following rounds the choice is made by the vote of five trees built with the labels seen up to that point. ```python # reinforcement.py — the label is seen only once a visit is made. Six rounds of twenty # visits each; every round is chosen by the vote of five trees built with what has been seen so far. ROUNDS, STEP = 6, 20 r = rng(SEED + 4404) POOL = sorted(TRAIN, key=lambda a: r()) visited, seen = [], set() print(fmt(["round", "visits", "found", "total"])) for t in range(ROUNDS): if not visited: chosen = POOL[:STEP] else: trees = [fit([visited[int(r() * len(visited))] for _ in visited], 3, 10) for _ in range(5)] chosen = sorted((a for a in POOL if a["subscriber_id"] not in seen), key=lambda a: (-sum(apply(d, a) for d in trees), a["subscriber_id"]))[:STEP] seen.update(a["subscriber_id"] for a in chosen) visited += chosen print(fmt([t + 1, STEP, positives(chosen), positives(visited)])) tree_rl, tree_full = fit(visited, 3, 20), fit(TRAIN) print(f"\nillicit cases found in {len(visited)} visits: {positives(visited)}; " f"if the same budget were spent at random: {positives(POOL[:ROUNDS * STEP])}") print(fmt(["setup", "labels", "train", "held-out"])) FINAL = [("supervised", 883, lambda a: apply(tree_full, a)), ("unsupervised", 6, CLUSTERS[6][1]), ("semi-supervised", 60, f_semi), ("reinforcement", ROUNDS * STEP, lambda a: apply(tree_rl, a))] for setup, n, f in FINAL: print(fmt([setup, n, f"{accuracy(f, TRAIN):.4f}", f"{accuracy(f, HELD_OUT):.4f}"])) ``` ``` round visits found total 1 20 4 4 2 20 11 15 3 20 8 23 4 20 11 34 5 20 19 53 6 20 20 73 illicit cases found in 120 visits: 73; if the same budget were spent at random: 25 setup labels train held-out supervised 883 0.8709 0.8594 unsupervised 6 0.7780 0.8117 semi-supervised 60 0.7780 0.8090 reinforcement 120 0.7973 0.8090 ``` The found column fluctuates from round to round — 4, 11, 8, 11, 19, 20 — but the return, that is the total, rises by accumulating: 4, 15, 23, 34, 53, 73. In the sixth round all twenty of the twenty subscribers chosen are illicit cases. In total 73 illicit cases were found in 120 visits; had the same budget been spent at random, 25 would have been found. **The baseline here is not the most frequent class but the random visit**, because what is measured is not classification accuracy but the return on the budget, and the random visit is the counterpart of the non-learning procedure on this measure. The difference is 48 illicit cases. The last table puts the four setups side by side. In the accuracy column three setups stand very close — 0.8117, 0.8090, 0.8090 — but their label columns are 6, 60 and 120. Producing the same number with a budget twenty times different is not the same thing. Of the 120 labels of the reinforcement setup, 73 came back as illicit cases found in the field; its number is read on the return row. #### Summary - The four setups solve the same problem and all four are measured on the same held-out set: supervised 0.8594 with 883 labels, unsupervised 0.8117 with 6 labels, semi-supervised 0.8090 with 60 labels, reinforcement 0.8090 with 120 visits; the baseline is 0.7533. - As the label budget shrinks, the difference over the baseline erodes: 0.1061 at 883 labels, 0.0886 at 400, 0.0212 at 30. In the same fall the training figure rises in the opposite direction, from 0.8709 to 0.9400. - The output of the unsupervised setup is not a decision but a structure; turning the structure into a decision asks for one visit per cluster. The two-cluster split sits exactly on the baseline, the six-cluster split gains 0.0584. - In the semi-supervised setup the 823 unlabeled rows carry the model built with the same 60 labels from 0.7905 to 0.8090; only 76.91 percent of the pseudo-labels are correct. - The measure of the reinforcement setup is the return: 73 illicit cases in 120 visits, 25 with random visits; the difference is 48 illicit cases, and this number does not appear in the accuracy column. #### Next Step Three of the four setups paid for labels and the fourth earned them with visits. None of them asked this: can the label be **derived from the data**. In the readings table there is a structure that no one labeled — the path each subscriber follows across the periods. Predicting the next period from the reading of one period asks for no label, because the answer is already in the table. The next lesson builds this setup: a representation is learned through an unlabeled pretext task, that representation is given to the main task as features, and it is measured how many points the pretext task adds over the baseline in the main task. ### Self-Supervised Learning Source: https://academia.sh/en/courses/introduction-to-machine-learning/self-supervised-learning The label is derived from the data: the last reading of each subscriber is hidden and predicted from the earlier readings. On the pretext task the baseline gives a mean absolute error of 6.3776 m³ on the held-out set and the least squares model 5.9661 m³ — a gain of 0.4115 m³, obtained with zero labels. When the three numbers produced by the pretext task are carried to the main task, the 0.7851 measured with the record features alone rises to 0.8037. When the same representation is built from the baseline of the pretext task, it gives 0.8064, so the contribution of the learned part is minus 0.0027: the difference comes not from what the pretext task learned but from the raw reading that was carried along. The nine hand-derived features still give 0.8594. The previous lesson counted how many labels the four setups ask for. Three of them paid for labels and the fourth earned them with visits. None of them asked this: can the label be **derived from the data**. In the readings table there is a structure that no one labeled — the trace each subscriber leaves across the periods. Closing one period's reading and predicting it from the rest asks for no label, because the answer is already in the table. This lesson sets up that setup and shows that two separate tasks have two separate baselines. - **FC14. Self-supervised learning**: the label is not given from outside, it is derived from the data itself. The procedure works as supervised, but the supervision is provided by the data. - **FC15. Pretext task**: a task whose answer is already in the data. The aim is not to solve that task but to learn something while solving it. - **FC16. Representation**: the numbers that the pretext task produces and that are carried to the main task as features. In this lesson three numbers are carried. - **FC17.** The pretext task has **its own baseline**, and that baseline produces a representation too. Unless the two representations are compared in the same place, the contribution of the pretext task is not counted as measured. - **FC18.** The pretext task predicts a numerical value; this kind of task is called **regression**. The term is kept apart from the regression used in the Statistical Analysis and Exploratory Data Analysis course to model the relationship between variables: here the aim is not to explain a relationship but to predict an unseen value. - **FC19.** The measure of regression is the **mean absolute error**, and that of classification is accuracy. This lesson uses two measures and chooses the baseline of each separately. #### Rebuilding the Fictional Dataset ```python # setup.py — the fictional dataset of lesson 01 is rebuilt with the same seed: the readings # table, the subscriber summary, the fictional label, the train / held-out split and the tree learner. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] FEATURES = ["mean", "per_person", "drop", "zero_readings", "reading_count", "satisfaction", "household_size", "highest", "lowest"] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 _subscribers, _dropped, _series = [], [], {} for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] k = {"subscriber_id": 10001 + i, "region": b[0], "base": b[2], "household_size": pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]), "satisfaction": pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) + 1} (_dropped if r() < 0.046 else _subscribers).append(k) _valid = {a["subscriber_id"] for a in _subscribers} for k in _subscribers + _dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor(k["base"] * math.exp( (r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] in _valid: _series.setdefault(k["subscriber_id"], []).append(t) r() SUBSCRIBERS = [] for k in _subscribers: t = _series.get(k["subscriber_id"]) if not t: continue mean = sum(t) / len(t) a = {"subscriber_id": k["subscriber_id"], "region": k["region"], "series": t, "household_size": k["household_size"], "satisfaction": k["satisfaction"], "reading_count": len(t), "mean": mean, "highest": max(t), "lowest": min(t), "zero_readings": sum(1 for v in t if v == 0), "per_person": mean / max(1, k["household_size"]), "drop": (t[0] - t[-1]) / t[0] if len(t) > 1 and t[0] > 0 else 0.0} r = rng(SEED + 31000 + a["subscriber_id"]) a["suspicion"] = 1 if (1.30 * a["zero_readings"] + 1.25 * (a["drop"] > 0.45) + 1.05 * (a["per_person"] > 12.0) + 0.65 * (a["reading_count"] < 3) + 0.45 * (a["satisfaction"] <= 2) - 1.55 + (r() + r() - 1.0) * 1.1) > 0 else 0 SUBSCRIBERS.append(a) r = rng(SEED + 90001) TRAIN, HELD_OUT = [], [] for a in SUBSCRIBERS: (TRAIN if r() < 0.70 else HELD_OUT).append(a) positives = lambda k: sum(a["suspicion"] for a in k) accuracy = lambda f, k: sum(1 for a in k if f(a) == a["suspicion"]) / len(k) _impurity = lambda k: 2 * (positives(k) / len(k)) * (1 - positives(k) / len(k)) if k else 0.0 def fit(k, depth=3, min_size=40): """The greedy decision tree of lesson 01: the split criterion and stopping rule are there.""" best, choice = _impurity(k) * len(k), None if depth and len(k) >= min_size: for feat in FEATURES: for threshold in sorted({a[feat] for a in k})[1:]: left = [a for a in k if a[feat] < threshold] right = [a for a in k if a[feat] >= threshold] m = _impurity(left) * len(left) + _impurity(right) * len(right) if m < best - 1e-9: best, choice = m, (feat, threshold, left, right) if choice is None: return 1 if 2 * positives(k) > len(k) else 0 lt, rt = fit(choice[2], depth - 1, min_size), fit(choice[3], depth - 1, min_size) return lt if lt == rt else (choice[0], choice[1], lt, rt) def apply(d, a): while isinstance(d, tuple): d = d[2] if a[d[0]] < d[1] else d[3] return d print(f"fictional dataset (seed {SEED}): {len(SUBSCRIBERS)} subscribers, {len(TRAIN)} train, " f"{len(HELD_OUT)} held-out, {positives(SUBSCRIBERS)} suspicious") ``` ``` fictional dataset (seed 20260218): 1260 subscribers, 883 train, 377 held-out, 327 suspicious ``` #### The Pretext Task and Its Own Baseline The pretext task is this: the last reading of each subscriber is hidden and predicted from the earlier readings, the number of readings and the record information. This task requires no door visit; the answer is already in the table, it is just not shown to the model. Because the task predicts a numerical value, its baseline must be numerical too. The non-learning procedure here is "telling everyone the overall mean of the set"; this is the counterpart in regression of the most frequent class in classification. Least squares is used on the model side and the normal equations are solved by elimination. ```python # pretext.py — an unlabeled task: the last reading of each subscriber is hidden and predicted # from the earlier readings. The answer is already in the table, no label is spent. import statistics from collections import Counter def solve(A, b): """Linear system of equations: elimination with partial pivoting.""" n = len(b) M = [A[i][:] + [b[i]] for i in range(n)] for i in range(n): p = max(range(i, n), key=lambda s: abs(M[s][i])) M[i], M[p] = M[p], M[i] if abs(M[i][i]) < 1e-12: # singular column: the coefficient is left at zero continue for s in range(n): if s != i and M[s][i]: f = M[s][i] / M[i][i] for c in range(i, n + 1): M[s][c] -= f * M[i][c] return [M[i][n] / M[i][i] if abs(M[i][i]) > 1e-12 else 0.0 for i in range(n)] def least_squares(X, y): k = len(X[0]) return solve([[sum(x[i] * x[j] for x in X) for j in range(k)] for i in range(k)], [sum(x[i] * t for x, t in zip(X, y)) for i in range(k)]) OVERALL = statistics.fmean(v for a in TRAIN for v in a["series"]) def inputs(a): previous = a["series"][:-1] return [1.0, statistics.fmean(previous) if previous else OVERALL, float(len(a["series"])), float(a["household_size"]), float(a["satisfaction"])] target = lambda k: [a["series"][-1] for a in k] WEIGHTS = least_squares([inputs(a) for a in TRAIN], target(TRAIN)) predict_value = lambda a: sum(c * v for c, v in zip(WEIGHTS, inputs(a))) error = lambda f, k: statistics.fmean(abs(f(a) - a["series"][-1]) for a in k) W = [34, 12, 12] fmt = lambda h: str(h[0]).ljust(W[0]) + str(h[1]).rjust(W[1]) + str(h[2]).rjust(W[2]) print("pretext task: prediction of the last reading, measure mean absolute error (m3)") print(fmt(["procedure", "train", "held-out"])) print(fmt(["baseline: always the overall mean", f"{error(lambda a: OVERALL, TRAIN):.4f}", f"{error(lambda a: OVERALL, HELD_OUT):.4f}"])) print(fmt(["least squares", f"{error(predict_value, TRAIN):.4f}", f"{error(predict_value, HELD_OUT):.4f}"])) print("coefficients", [round(v, 4) for v in WEIGHTS]) ``` ``` pretext task: prediction of the last reading, measure mean absolute error (m3) procedure train held-out baseline: always the overall mean 5.9664 6.3776 least squares 5.6979 5.9661 coefficients [18.2859, 0.262, -1.5773, -0.0497, -0.1704] ``` The pretext task beats the baseline on its own measure: on the held-out set the error falls from 6.3776 m³ to 5.9661 m³, a gain of 0.4115 m³. This gain was obtained with **zero labels**; the only thing spent is the table itself. The coefficients tell what the model learned. The coefficient of the mean of the earlier readings is 0.2620: a subscriber's past consumption partly determines the last reading, but weakly. The coefficient of the number of readings is -1.5773; the last reading of subscribers with all three periods is systematically lower, because in the fictional dataset each period falls below the one before. The coefficients of household size and satisfaction are close to zero. #### The Contribution of the Representation to the Main Task What is carried from the pretext task to the main task is not the prediction itself but the relationship between the prediction and the actual value. Three numbers are carried: the value the model expected, the difference between it and the actual value, and the ratio of that difference to the expected value. So that the comparison is honest, the main task is this time built with **a narrow feature set**: only the three columns that come from the subscriber record. This is the only information that can be obtained without touching the readings table; the contribution of the representation is read as the difference added on top of this narrowness. ```python # representation.py — the three numbers derived by the pretext task are carried to the main task # as features. The representation was learned with zero labels; the main task uses 883 labels. for a in SUBSCRIBERS: b = predict_value(a) a["expected"], a["deviation"] = b, a["series"][-1] - b a["relative_deviation"] = (a["series"][-1] - b) / (abs(b) + 1.0) RECORD = ["household_size", "satisfaction", "reading_count"] REPRESENTATION = ["expected", "deviation", "relative_deviation"] HAND = FEATURES[:] most_frequent = Counter(a["suspicion"] for a in TRAIN).most_common(1)[0][0] BASE_HELD = accuracy(lambda a: most_frequent, HELD_OUT) def measure(name, feats): global FEATURES FEATURES = feats d = fit(TRAIN) inside, outside = accuracy(lambda a: apply(d, a), TRAIN), accuracy( lambda a: apply(d, a), HELD_OUT) FEATURES = HAND print(str(name).ljust(38) + str(len(feats)).rjust(7) + f"{inside:.4f}".rjust(12) + f"{outside:.4f}".rjust(12) + f"{outside - BASE_HELD:+.4f}".rjust(13)) print("main task: illicit-use suspicion, measure accuracy") print("feature set".ljust(38) + "count".rjust(7) + "train".rjust(12) + "held-out".rjust(12) + "vs baseline".rjust(13)) print("baseline: always most frequent class".ljust(38) + "0".rjust(7) + f"{accuracy(lambda a: most_frequent, TRAIN):.4f}".rjust(12) + f"{BASE_HELD:.4f}".rjust(12) + "+0.0000".rjust(13)) measure("record only", RECORD) measure("record + representation", RECORD + REPRESENTATION) measure("nine hand-derived features", HAND) ``` ``` main task: illicit-use suspicion, measure accuracy feature set count train held-out vs baseline baseline: always most frequent class 0 0.7350 0.7533 +0.0000 record only 3 0.7939 0.7851 +0.0318 record + representation 6 0.8313 0.8037 +0.0504 nine hand-derived features 9 0.8709 0.8594 +0.1061 ``` The three rows represent three separate sources of information. With the subscriber record alone, the result goes 0.0318 above the baseline. When the three numbers produced by the pretext task are added, the difference rises to 0.0504; the contribution is 0.0186 and no label was spent on it. The nine hand-derived features still give 0.1061 — self-supervision has not replaced domain knowledge, it has closed a quarter of the gap. The label accounting is written down too. The model of the pretext task was built only with the readings of the training set; not a single row of the 377 subscribers in the held-out set entered the computation of the coefficients. In the main task all three rows use the same 883 labels, which means the difference between the rows comes not from the number of labels but only from the feature set. The cost of the representation is zero labels, and this is what separates it from the nine hand-derived features: to write those nine columns, someone had to look at the consumption series and decide which summary would be useful. #### Where the Contribution Comes From The table read up to here seems to say this: the pretext task learned something and what it learned was carried to the main task. This claim can be tested, because the baseline of the pretext task produces a representation too. The overall mean is put in place of the prediction, the same three numbers are computed again and they are measured in the same place. This representation, which carries nothing learned, shows how much of what was learned really comes from learning. ```python # audit.py — the baseline of the pretext task produces a representation too: the overall mean # is put in place of the prediction. The two representations are compared in the same place. for a in SUBSCRIBERS: a["expected_base"], a["deviation_base"] = OVERALL, a["series"][-1] - OVERALL a["relative_deviation_base"] = (a["series"][-1] - OVERALL) / (OVERALL + 1.0) BASE_REPRESENTATION = ["expected_base", "deviation_base", "relative_deviation_base"] print("feature set".ljust(38) + "count".rjust(7) + "train".rjust(12) + "held-out".rjust(12) + "vs baseline".rjust(13)) measure("record", RECORD) measure("record + baseline representation", RECORD + BASE_REPRESENTATION) measure("record + learned representation", RECORD + REPRESENTATION) ``` ``` feature set count train held-out vs baseline record 3 0.7939 0.7851 +0.0318 record + baseline representation 6 0.8301 0.8064 +0.0531 record + learned representation 6 0.8313 0.8037 +0.0504 ``` The claim collapsed. The representation built from the baseline gives 0.8064, the learned representation gives 0.8037; the contribution of the learned part is **minus 0.0027**. What really contributes to the main task is not the relationship the pretext task learned but the **raw observation** that the representation carries along. Because the overall mean is constant, the three columns in the second row are in fact three different ways of writing the last reading, and giving the decision tree the last reading as a new column explains the whole contribution. The consequence of this is at the curriculum level. The 0.4115 m³ gained on the pretext task is not the number gained on the main task. The two tasks have two separate measures, two separate baselines and two separate results; beating the baseline in one does not mean beating it in the other. To say that a pretext task works, one must write not the gain on its own measure but **the difference that it adds in the main task over the representation of the baseline**. It can also be written why the fictional dataset behaves this way. In this dataset the periods of the same subscriber are generated largely independently of one another; the reading of one period weakly determines the next, and the coefficient of the earlier mean stays at 0.2620. In a dataset strongly tied over time, the same pretext task would give a different number. What is measured is not the method itself but the contribution of the method **on this data**. #### Summary - In the self-supervised setup the label does not come from outside, it is derived from the data: the last reading is hidden and predicted from the earlier readings. - The pretext task beats the baseline on its own measure: on the held-out set the mean absolute error falls from 6.3776 m³ to 5.9661 m³, a gain of 0.4115 m³ with zero labels spent. - When the three numbers produced by the pretext task are carried to the main task, the 0.7851 measured with the record features alone rises to 0.8037; the nine hand-derived features give 0.8594. - When the same representation is built from the **baseline** of the pretext task, it gives 0.8064: the contribution of the learned part is minus 0.0027 and the difference comes from the raw observation carried along. - Beating the baseline in the pretext task does not mean beating the baseline in the main task; the two tasks have two separate measures and two separate baselines. #### Next Step In three lessons a single question was asked: is this subscriber suspicious. Because the question is binary, the baseline also stayed the same throughout — telling the most frequent class. Yet the same table can carry other questions too: how many cubic meters this subscriber will consume next period, into how many natural groups the subscribers split, which hundred subscribers the team going out to the field today should visit and in what order. The next lesson asks the same data four separate questions and chooses the baseline of each question separately: four separate non-learning procedures are written for classification, regression, clustering and ranking, the reasons for their choice are recorded and it is shown that the four baselines give four separate numbers. ### Application Areas Source: https://academia.sh/en/courses/introduction-to-machine-learning/application-areas Four questions are asked of the same fictional subscriber table and four separate baselines are chosen: in classification the most frequent class gives 0.7533 on the held-out set and the tree 0.8594; in regression the overall mean gives a mean absolute error of 5.2255 m³ and least squares 4.1275 m³; in clustering a random four-way grouping gives a within-cluster distance of 2.7683 and the four learned centers 2.1551; in ranking a random order finds 20 illicit cases in the first 100 visits, an order starting from the highest consumption finds 30, and an ensemble of nine trees finds 71. Of the four measures, a large number is good in two and a small number is good in two; the numbers can be compared not with one another but only with their own baselines. In three lessons a single question was asked: is this subscriber suspicious. Because the question is binary, the baseline also stayed the same throughout — telling the most frequent class. The same table can carry other questions too. How many cubic meters will this subscriber consume next period. Into how many natural groups do the subscribers split. Which hundred subscribers should the team going out to the field today visit, and in what order. This lesson asks the same data four questions. The only thing that does not change during the lesson is the table; what changes is the question, the measure of the question and **the baseline of the question**. All four are measured on the same held-out set. - **FC20.** A dataset does not define a single problem. What determines the problem is not the data but the question asked; which column is the label in the same table changes together with the question. - **FC21. Classification**: the output is a value from a finite set of labels. Its baseline is the **most frequent class**, because it is the best single decision that can be given without looking at anything. - **FC22. Regression**: the output is a numerical value. Its baseline is the **overall mean**; if the mean absolute error is measured, the median is a better baseline, and the choice is written down. - **FC23. Clustering**: there is no label, the output is a grouping. Its baseline is **random grouping**, because it produces the same number of groups and uses no structure. - **FC24. Ranking**: the output is not a single decision but an **order**, and the measure looks at the budget at the head of the order. Its baseline is the **random order**; a non-learning heuristic order is put next to it too, because that is the procedure actually used in the field. - **FC25.** Of the four measures, a large number is good in two and a small number is good in two. The numbers of the models can be compared not with one another but only with their own baselines. #### Rebuilding the Fictional Dataset ```python # setup.py — the fictional dataset of lesson 01 is rebuilt with the same seed: the readings # table, the subscriber summary, the fictional label, the train / held-out split and the tree learner. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] FEATURES = ["mean", "per_person", "drop", "zero_readings", "reading_count", "satisfaction", "household_size", "highest", "lowest"] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 _subscribers, _dropped, _series = [], [], {} for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] k = {"subscriber_id": 10001 + i, "region": b[0], "base": b[2], "household_size": pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]), "satisfaction": pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) + 1} (_dropped if r() < 0.046 else _subscribers).append(k) _valid = {a["subscriber_id"] for a in _subscribers} for k in _subscribers + _dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor(k["base"] * math.exp( (r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] in _valid: _series.setdefault(k["subscriber_id"], []).append(t) r() SUBSCRIBERS = [] for k in _subscribers: t = _series.get(k["subscriber_id"]) if not t: continue mean = sum(t) / len(t) a = {"subscriber_id": k["subscriber_id"], "region": k["region"], "series": t, "household_size": k["household_size"], "satisfaction": k["satisfaction"], "reading_count": len(t), "mean": mean, "highest": max(t), "lowest": min(t), "zero_readings": sum(1 for v in t if v == 0), "per_person": mean / max(1, k["household_size"]), "drop": (t[0] - t[-1]) / t[0] if len(t) > 1 and t[0] > 0 else 0.0} r = rng(SEED + 31000 + a["subscriber_id"]) a["suspicion"] = 1 if (1.30 * a["zero_readings"] + 1.25 * (a["drop"] > 0.45) + 1.05 * (a["per_person"] > 12.0) + 0.65 * (a["reading_count"] < 3) + 0.45 * (a["satisfaction"] <= 2) - 1.55 + (r() + r() - 1.0) * 1.1) > 0 else 0 SUBSCRIBERS.append(a) r = rng(SEED + 90001) TRAIN, HELD_OUT = [], [] for a in SUBSCRIBERS: (TRAIN if r() < 0.70 else HELD_OUT).append(a) positives = lambda k: sum(a["suspicion"] for a in k) accuracy = lambda f, k: sum(1 for a in k if f(a) == a["suspicion"]) / len(k) _impurity = lambda k: 2 * (positives(k) / len(k)) * (1 - positives(k) / len(k)) if k else 0.0 def fit(k, depth=3, min_size=40): """The greedy decision tree of lesson 01: the split criterion and stopping rule are there.""" best, choice = _impurity(k) * len(k), None if depth and len(k) >= min_size: for feat in FEATURES: for threshold in sorted({a[feat] for a in k})[1:]: left = [a for a in k if a[feat] < threshold] right = [a for a in k if a[feat] >= threshold] m = _impurity(left) * len(left) + _impurity(right) * len(right) if m < best - 1e-9: best, choice = m, (feat, threshold, left, right) if choice is None: return 1 if 2 * positives(k) > len(k) else 0 lt, rt = fit(choice[2], depth - 1, min_size), fit(choice[3], depth - 1, min_size) return lt if lt == rt else (choice[0], choice[1], lt, rt) def apply(d, a): while isinstance(d, tuple): d = d[2] if a[d[0]] < d[1] else d[3] return d print(f"fictional dataset (seed {SEED}): {len(SUBSCRIBERS)} subscribers, {len(TRAIN)} train, " f"{len(HELD_OUT)} held-out, {positives(SUBSCRIBERS)} suspicious") ``` ``` fictional dataset (seed 20260218): 1260 subscribers, 883 train, 377 held-out, 327 suspicious ``` #### A Binary Decision and a Numerical Value The first two questions are labeled and differ from each other only in the type of the output. The first asks for a class: is this subscriber suspicious. The second asks for a number: how many cubic meters does this subscriber consume on average. The second question is not for the field team but for network planning, and it must be answerable before the reading arrives; this is why only the columns that come from the subscriber record, together with the region, are given to the regression model. The baselines are chosen separately because the measures are separate. For accuracy, the most frequent class; for the mean absolute error, the overall mean. The second choice is debatable: if the mean absolute error is measured, the best single non-learning number is not the mean but the median. Here the mean was chosen, because squared error will also be used in later courses and a single baseline will be kept across the two measures; because the choice is debatable, it is written down. ```python # two_problems.py — two questions to the same table: a binary decision and a numerical value. # The baseline of each question is chosen separately; the most frequent class for the binary # question, the overall mean for the numerical one, because the measures are different. import statistics from collections import Counter RESULTS = [] HEADER = ["problem", "measure", "base train", "base held-out", "model train", "model held-out"] WIDTHS = [16, 40, 15, 15, 15, 15] fmt = lambda h: str(h[0]).ljust(WIDTHS[0]) + str(h[1]).ljust(WIDTHS[1]) + "".join( str(v).rjust(WIDTHS[i + 2]) for i, v in enumerate(h[2:])) def record(name, measure, bt, bh, mt, mh, form="{:.4f}"): RESULTS.append([name, measure] + [form.format(v) for v in (bt, bh, mt, mh)]) print(fmt(RESULTS[-1])) most_frequent = Counter(a["suspicion"] for a in TRAIN).most_common(1)[0][0] TREE = fit(TRAIN) REGION_NAMES = [b[0] for b in REGION] def regression_inputs(a): return ([1.0, float(a["household_size"]), float(a["satisfaction"]), float(a["reading_count"])] + [1.0 if a["region"] == b else 0.0 for b in REGION_NAMES[1:]]) def solve(A, b): n = len(b) M = [A[i][:] + [b[i]] for i in range(n)] for i in range(n): p = max(range(i, n), key=lambda s: abs(M[s][i])) M[i], M[p] = M[p], M[i] if abs(M[i][i]) < 1e-12: continue for s in range(n): if s != i and M[s][i]: f = M[s][i] / M[i][i] for c in range(i, n + 1): M[s][c] -= f * M[i][c] return [M[i][n] / M[i][i] if abs(M[i][i]) > 1e-12 else 0.0 for i in range(n)] k = len(regression_inputs(TRAIN[0])) X, t = [regression_inputs(a) for a in TRAIN], [a["mean"] for a in TRAIN] WEIGHTS = solve([[sum(x[i] * x[j] for x in X) for j in range(k)] for i in range(k)], [sum(x[i] * v for x, v in zip(X, t)) for i in range(k)]) OVERALL = statistics.fmean(t) predict_value = lambda a: sum(c * v for c, v in zip(WEIGHTS, regression_inputs(a))) error = lambda f, s: statistics.fmean(abs(f(a) - a["mean"]) for a in s) print(fmt(HEADER)) record("classification", "accuracy (high is good)", accuracy(lambda a: most_frequent, TRAIN), accuracy(lambda a: most_frequent, HELD_OUT), accuracy(lambda a: apply(TREE, a), TRAIN), accuracy(lambda a: apply(TREE, a), HELD_OUT)) record("regression", "mean absolute error m3 (low is good)", error(lambda a: OVERALL, TRAIN), error(lambda a: OVERALL, HELD_OUT), error(predict_value, TRAIN), error(predict_value, HELD_OUT)) ``` ``` problem measure base train base held-out model train model held-out classification accuracy (high is good) 0.7350 0.7533 0.8709 0.8594 regression mean absolute error m3 (low is good) 4.8767 5.2255 3.7226 4.1275 ``` The two rows are read in two different directions. In classification the model is **above** the baseline by 0.1061; in regression it is **below** it by 1.0980 m³, and that is what is good. The magnitudes of the numbers cannot be compared either: 0.8594 is unitless, 4.1275 is cubic meters. #### Unlabeled Grouping and an Order The third question uses no label at all: into how many natural groups do the subscribers split. The measure is the mean distance of each subscriber to the center of its own group; a small value shows that the groups are tight. The non-learning procedure here distributes the subscribers into **four random groups** and computes the center of each group from its own members; in this way the same number of groups is produced but no structure is used. The fourth question asks for an order. The daily capacity of the field team is a hundred visits, and the measure is how many illicit cases are found in the first hundred of the order. Two baselines are written here. The random order is the non-learning procedure itself; the order starting from the highest mean consumption is the heuristic procedure actually used in the field, and the model has to beat that too. ```python # two_more_problems.py — the third question is unlabeled, the fourth asks for an order, not # a number. The baseline of the third is random grouping, that of the fourth a random order. MEANS = {o: statistics.fmean(a[o] for a in TRAIN) for o in FEATURES} STDS = {o: statistics.pstdev(a[o] for a in TRAIN) or 1.0 for o in FEATURES} vec = lambda a: [(a[o] - MEANS[o]) / STDS[o] for o in FEATURES] dist = lambda v, m: math.sqrt(sum((x - y) ** 2 for x, y in zip(v, m))) within_dist = lambda s, M: statistics.fmean(min(dist(vec(a), m) for m in M) for a in s) K = 4 CENTERS = [vec(a) for a in TRAIN[:K]] for _ in range(25): groups = [[] for _ in range(K)] for a in TRAIN: groups[min(range(K), key=lambda i: dist(vec(a), CENTERS[i]))].append(vec(a)) for i in range(K): if groups[i]: CENTERS[i] = [sum(c) / len(c) for c in zip(*groups[i])] r = rng(SEED + 6104) random_groups = [[] for _ in range(K)] for a in TRAIN: random_groups[int(r() * K)].append(vec(a)) RANDOM_CENTERS = [[sum(c) / len(c) for c in zip(*g)] for g in random_groups] TOP = 100 r = rng(SEED + 6205) ENSEMBLE = [fit([TRAIN[int(r() * len(TRAIN))] for _ in TRAIN]) for _ in range(9)] score = lambda a: sum(apply(d, a) for d in ENSEMBLE) r = rng(SEED + 6206) lots = {a["subscriber_id"]: r() for a in SUBSCRIBERS} found_in_top = lambda f, s: positives(sorted(s, key=f)[:TOP]) print(fmt(HEADER)) record("clustering", "within-cluster distance (low is good)", within_dist(TRAIN, RANDOM_CENTERS), within_dist(HELD_OUT, RANDOM_CENTERS), within_dist(TRAIN, CENTERS), within_dist(HELD_OUT, CENTERS)) record("ranking", f"illicit cases in top {TOP} (high is good)", found_in_top(lambda a: lots[a["subscriber_id"]], TRAIN), found_in_top(lambda a: lots[a["subscriber_id"]], HELD_OUT), found_in_top(lambda a: (-score(a), a["subscriber_id"]), TRAIN), found_in_top(lambda a: (-score(a), a["subscriber_id"]), HELD_OUT), "{:.0f}") print(f"\nsecond baseline in ranking — start from the highest mean consumption: " f"train {found_in_top(lambda a: (-a['mean'], a['subscriber_id']), TRAIN)}, " f"held-out {found_in_top(lambda a: (-a['mean'], a['subscriber_id']), HELD_OUT)}") print(f"for comparison: train {positives(TRAIN)} suspicious / {len(TRAIN)} subscribers, " f"held-out {positives(HELD_OUT)} suspicious / {len(HELD_OUT)} subscribers") ``` ``` problem measure base train base held-out model train model held-out clustering within-cluster distance (low is good) 2.7237 2.7683 2.1678 2.1551 ranking illicit cases in top 100 (high is good) 29 20 86 71 second baseline in ranking — start from the highest mean consumption: train 38, held-out 30 for comparison: train 234 suspicious / 883 subscribers, held-out 93 suspicious / 377 subscribers ``` In clustering the four learned centers bring the distance of 2.7683 of the random four groups down to 2.1551; the gain is 0.6132. A warning stands beside this number: the measure of clustering never looks at the label, which means a "good" clustering is not a clustering that finds illicit cases. In the previous lesson it was measured separately how close a six-cluster split came to the baseline on the illicit-use question; these two numbers are not two measures of the same thing but two separate measures of two separate problems. In ranking three numbers stand side by side. The random order finds 20 illicit cases in the first hundred visits of the held-out set — which is what is expected, given that 93 of the 377 subscribers are suspicious. The heuristic order starting from the highest consumption finds 30; domain knowledge beats random by 10 illicit cases. The order of the ensemble of nine trees finds 71. The model is 41 illicit cases above the heuristic and 51 above random, and **depending on which baseline is written, the gain of the model is 41 or 51**. Choosing the baseline is a measurement decision. #### Four Problems, Four Baselines ```python # four_baselines.py — four problems, four separate baselines, four separate measures and two # separate directions. print(fmt(HEADER)) for s in RESULTS: print(fmt(s)) ``` ``` problem measure base train base held-out model train model held-out classification accuracy (high is good) 0.7350 0.7533 0.8709 0.8594 regression mean absolute error m3 (low is good) 4.8767 5.2255 3.7226 4.1275 clustering within-cluster distance (low is good) 2.7237 2.7683 2.1678 2.1551 ranking illicit cases in top 100 (high is good) 29 20 86 71 ``` The table came out of a single dataset and gave four separate rows. The model columns cannot be compared with one another: putting 0.8594 and 4.1275 side by side and counting one of them as larger means nothing, because one is a ratio and the other is cubic meters, and in one a large number is good while in the other it is bad. The only comparable thing is the difference between the two columns within each row. The baseline columns carry information too. In classification the baseline is already 0.7533; the largest share the model could gain is 0.2467, and 0.1061 is 43.0 percent of that. In ranking the baseline is only 20 and the ceiling is 93; with 71 the model has closed 69.9 percent of the gap. The same model, the same data, the same day — but two separate success stories on two questions. #### Summary - A dataset does not define a single problem; the same fictional subscriber table carries all four of the classification, regression, clustering and ranking questions. - The four questions have four separate baselines: the most frequent class 0.7533, the overall mean 5.2255 m³, a random four-way grouping 2.7683 within-cluster distance, a random order 20 illicit cases in the first 100 visits. - The held-out numbers of the models are 0.8594, 4.1275 m³, 2.1551 and 71 illicit cases respectively; a large number is good in two and a small number is good in two. - A second baseline was written in ranking: the heuristic order starting from the highest consumption finds 30 illicit cases, so the gain of the model is 41 or 51 illicit cases depending on the baseline written. - The numbers of the models cannot be compared with one another; the only comparable thing is the distance of each row to its own baseline. #### Next Step All four problems were read from a single point: a ready table, a ready feature set, a ready split and a single measurement. These lessons showed how a model is built, not how a job runs. In reality the problem definition changes after measurement, the feature set widens after evaluation, the threshold is moved after deployment, and the flow never runs once from start to finish. The next lesson sets up this flow as seven steps and what it measures is not the steps themselves but **the returns**: which step returns to which step is counted, and how much each return moved the held-out figure is printed. ### Machine Learning Workflow Source: https://academia.sh/en/courses/introduction-to-machine-learning/machine-learning-workflow The seven-step flow produces three returns, and the number each return moved is printed. The gate at the fifth step tests the gap over the baseline with a threshold of 0.0500: the model built with three columns gives 0.0318 and stays at the gate, with nine columns it gives 0.1061 and passes. At the sixth step the model flags 240 of 1,260 subscribers, the daily capacity is 40 visits and the list spreads over six days; the constraint turns the problem from classification into ranking, and the ranking finds 30 illicit cases in the first 40 visits, the random order 14. At the seventh step the measurement policy changes, the label does not change, yet the same model falls to 23 and returns to 30 when rebuilt on the drifted data. The seven-step flow is completed with a total of 22 step visits. Four lessons showed how a model is built: a ready table, a ready feature set, a ready split and a single measurement. In a real job none of these comes ready. The problem definition changes after measurement, the feature set widens after evaluation, and the flow never runs once from start to finish. This lesson builds the flow as seven steps and counts **not the steps but the returns**. The analysis workflow lesson in the Introduction to Data Analytics course had counted returns too; there the reason for a return was that a decision was seen only after the step it belonged to. Here what triggers the return is not a decision but a **measurement**: the flow goes back because a number stayed under a threshold. - **FC26.** The flow is seven steps: problem definition, data preparation, feature selection, model building, evaluation, deployment, monitoring. What is measured is not the number of steps but **the number of returns**. - **FC27. A gate** is the check that ties the decision to advance in the flow to a number. The threshold of a gate is written **before** the measurement; if it is written afterward, the threshold becomes not a decision but a justification. - **FC28.** In this run a return has three separate sources: the measurement failing to reach the threshold, the deployment constraint and drift of the data in monitoring. - **FC29.** The deployment constraint can change the problem definition. The output of classification is a **set**, that of ranking is a **queue**; what a team working with limited capacity wants is a queue. - **FC30.** Monitoring is the only step at which the number can change although the model never changes. What drifts is not the model itself but the data that reaches the model. - **FC31.** The decision the model produces after deployment is called **inference**. The term is distinct from the inference used in the Visualization, Reporting and Analysis at Scale course in the sense of the conclusion a reader draws from a chart: here the one making the inference is not the reader but the model. #### Rebuilding the Fictional Dataset ```python # setup.py — the fictional dataset of lesson 01 is rebuilt with the same seed: the readings # table, the subscriber summary, the fictional label, the train / held-out split and the tree learner. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] FEATURES = ["mean", "per_person", "drop", "zero_readings", "reading_count", "satisfaction", "household_size", "highest", "lowest"] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 _subscribers, _dropped, _series = [], [], {} for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] k = {"subscriber_id": 10001 + i, "region": b[0], "base": b[2], "household_size": pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]), "satisfaction": pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) + 1} (_dropped if r() < 0.046 else _subscribers).append(k) _valid = {a["subscriber_id"] for a in _subscribers} for k in _subscribers + _dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor(k["base"] * math.exp( (r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] in _valid: _series.setdefault(k["subscriber_id"], []).append(t) r() SUBSCRIBERS = [] for k in _subscribers: t = _series.get(k["subscriber_id"]) if not t: continue mean = sum(t) / len(t) a = {"subscriber_id": k["subscriber_id"], "region": k["region"], "series": t, "household_size": k["household_size"], "satisfaction": k["satisfaction"], "reading_count": len(t), "mean": mean, "highest": max(t), "lowest": min(t), "zero_readings": sum(1 for v in t if v == 0), "per_person": mean / max(1, k["household_size"]), "drop": (t[0] - t[-1]) / t[0] if len(t) > 1 and t[0] > 0 else 0.0} r = rng(SEED + 31000 + a["subscriber_id"]) a["suspicion"] = 1 if (1.30 * a["zero_readings"] + 1.25 * (a["drop"] > 0.45) + 1.05 * (a["per_person"] > 12.0) + 0.65 * (a["reading_count"] < 3) + 0.45 * (a["satisfaction"] <= 2) - 1.55 + (r() + r() - 1.0) * 1.1) > 0 else 0 SUBSCRIBERS.append(a) r = rng(SEED + 90001) TRAIN, HELD_OUT = [], [] for a in SUBSCRIBERS: (TRAIN if r() < 0.70 else HELD_OUT).append(a) positives = lambda k: sum(a["suspicion"] for a in k) accuracy = lambda f, k: sum(1 for a in k if f(a) == a["suspicion"]) / len(k) _impurity = lambda k: 2 * (positives(k) / len(k)) * (1 - positives(k) / len(k)) if k else 0.0 def fit(k, depth=3, min_size=40): """The greedy decision tree of lesson 01: the split criterion and stopping rule are there.""" best, choice = _impurity(k) * len(k), None if depth and len(k) >= min_size: for feat in FEATURES: for threshold in sorted({a[feat] for a in k})[1:]: left = [a for a in k if a[feat] < threshold] right = [a for a in k if a[feat] >= threshold] m = _impurity(left) * len(left) + _impurity(right) * len(right) if m < best - 1e-9: best, choice = m, (feat, threshold, left, right) if choice is None: return 1 if 2 * positives(k) > len(k) else 0 lt, rt = fit(choice[2], depth - 1, min_size), fit(choice[3], depth - 1, min_size) return lt if lt == rt else (choice[0], choice[1], lt, rt) def apply(d, a): while isinstance(d, tuple): d = d[2] if a[d[0]] < d[1] else d[3] return d print(f"fictional dataset (seed {SEED}): {len(SUBSCRIBERS)} subscribers, {len(TRAIN)} train, " f"{len(HELD_OUT)} held-out, {positives(SUBSCRIBERS)} suspicious") ``` ``` fictional dataset (seed 20260218): 1260 subscribers, 883 train, 377 held-out, 327 suspicious ``` #### Seven Steps and the First Gate The flow is defined at the first step: the question is "is this subscriber suspicious", the measure is accuracy, the baseline is the most frequent class. The threshold of the gate at the fifth step is also written here: the model must exceed the baseline on the held-out set by at least 0.0500. This number is an engineering decision, and it is a decision because it is written before the measurement; had it been written after the measurement it would have been a justification that legitimizes the result. At the second step the data is prepared. In the first pass the summaries of the readings table are not yet ready, so the third step chooses only the three columns that come from the subscriber record. ```python # workflow.py — seven steps and two gates. The gates look at a number: the gate at the fifth # step measures the gap over the baseline, the one at the sixth the daily field capacity. import statistics from collections import Counter STEPS = ["1 problem definition", "2 data preparation", "3 feature selection", "4 model building", "5 evaluation", "6 deployment", "7 monitoring"] ACCEPT_GAP, CAPACITY = 0.05, 40 RETURNS, FULL = [], FEATURES[:] RECORD = ["household_size", "satisfaction", "reading_count"] most_frequent = Counter(a["suspicion"] for a in TRAIN).most_common(1)[0][0] BASE_TRAIN = accuracy(lambda a: most_frequent, TRAIN) BASE_HELD = accuracy(lambda a: most_frequent, HELD_OUT) def tree_with(feats, subset): global FEATURES FEATURES = feats d = fit(subset) FEATURES = FULL return d W = [8, 16, 11, 11, 11, 11, 22] row = lambda h: str(h[0]).ljust(W[0]) + str(h[1]).ljust(W[1]) + "".join( str(v).rjust(W[i + 2]) for i, v in enumerate(h[2:6])) + " " + str(h[6]) print(row(["pass", "features", "baseline", "train", "held-out", "gap", "gate"])) TREES = {} for pass_no, feats in ((1, RECORD), (2, FULL)): d = TREES[pass_no] = tree_with(feats, TRAIN) inner, outer = accuracy(lambda a: apply(d, a), TRAIN), accuracy( lambda a: apply(d, a), HELD_OUT) gap = outer - BASE_HELD passed = gap >= ACCEPT_GAP print(row([pass_no, f"{len(feats)} columns", f"{BASE_HELD:.4f}", f"{inner:.4f}", f"{outer:.4f}", f"{gap:+.4f}", "passed" if passed else f"closed, threshold {ACCEPT_GAP:.2f}"])) if not passed: RETURNS.append([STEPS[4], STEPS[2], "gap below threshold", f"{gap:+.4f}", None]) else: RETURNS[-1][4] = f"{gap:+.4f}" flagged = sum(1 for a in SUBSCRIBERS if apply(TREES[2], a)) print(f"\nstep 6: the model flags {flagged} of {len(SUBSCRIBERS)} subscribers; " f"daily capacity is {CAPACITY} visits, so the list " f"spreads over {-(-flagged // CAPACITY)} days and has no order inside") RETURNS.append([STEPS[5], STEPS[0], "does not fit capacity", f"{flagged} flagged", None]) ``` ``` pass features baseline train held-out gap gate 1 3 columns 0.7533 0.7939 0.7851 +0.0318 closed, threshold 0.05 2 9 columns 0.7533 0.8709 0.8594 +0.1061 passed step 6: the model flags 240 of 1260 subscribers; daily capacity is 40 visits, so the list spreads over 6 days and has no order inside ``` The first pass stayed at the gate: 0.0318 is below the threshold. The flow returns from the fifth step to the third, and the feature set grows to nine columns with the summaries of the reading table. In the second pass the gap is 0.1061 and the gate opens. This is the number the return moved: 0.1061 instead of 0.0318. At the sixth step the second gate comes into play, and the model measures not performance but **the constraint of the job**. The model flags 240 of 1,260 subscribers. The daily capacity is 40 visits, so the list spreads over six days, and the output of classification does not say who will be visited first within the list. The model passed the gate on performance, it could not pass on the constraint. #### The Constraint Changes the Problem, Monitoring the Data From the sixth step the flow returns to the first. This is the longest return of the flow and it changes not the model but **the question**: a question that asks for a decision is replaced by a question that asks for an order. The measure changes too — the number of illicit cases found in the first 40 visits instead of accuracy — and the baseline is chosen anew: the random order. At the seventh step the model is in the field and is monitored. What monitoring measures is not the model itself, because the model does not change. What changes is the data that reaches the model: the network changes the reading policy, zero readings are filled with an estimate and missing periods are completed. The subscribers are the same, the true illicit-use status is the same, only the way it is written to the table has changed. ```python # ranking_and_monitoring.py — the problem is redefined: an order is wanted, not a decision. # After deployment the measurement policy changes and the same model is measured again. r = rng(SEED + 7205) ENSEMBLE = [fit([TRAIN[int(r() * len(TRAIN))] for _ in TRAIN]) for _ in range(9)] r = rng(SEED + 7206) lots = {a["subscriber_id"]: r() for a in SUBSCRIBERS} found_in_top = lambda f, s: positives(sorted(s, key=f)[:CAPACITY]) ranking = lambda T: (lambda a: (-sum(apply(d, a) for d in T), a["subscriber_id"])) random_order = lambda a: lots[a["subscriber_id"]] def drifted(a): """The reading policy changed: zero readings are filled with an estimate, missing periods are completed. The true illicit-use status, that is the label, does not change.""" filled = [v for v in a["series"] if v > 0] est = round(statistics.fmean(filled), 2) if filled else 0.0 t = [v if v > 0 else est for v in a["series"]] + [est] * (3 - len(a["series"])) avg = sum(t) / len(t) return {**a, "series": t, "reading_count": 3, "mean": avg, "highest": max(t), "lowest": min(t), "zero_readings": sum(1 for v in t if v == 0), "per_person": avg / max(1, a["household_size"]), "drop": (t[0] - t[-1]) / t[0] if t[0] > 0 else 0.0} DRIFT_TRAIN, DRIFT_HELD = [drifted(a) for a in TRAIN], [drifted(a) for a in HELD_OUT] r = rng(SEED + 7305) NEW = [fit([DRIFT_TRAIN[int(r() * len(DRIFT_TRAIN))] for _ in DRIFT_TRAIN]) for _ in range(9)] V = [30, 14, 12, 12] row2 = lambda h: str(h[0]).ljust(V[0]) + "".join(str(v).rjust(V[i + 1]) for i, v in enumerate(h[1:])) print(row2([f"illicit in first {CAPACITY} visits", "data", "train", "held-out"])) print(row2(["baseline: random order", "old", found_in_top(random_order, TRAIN), found_in_top(random_order, HELD_OUT)])) print(row2(["model: nine-tree order", "old", found_in_top(ranking(ENSEMBLE), TRAIN), found_in_top(ranking(ENSEMBLE), HELD_OUT)])) print(row2(["baseline: random order", "drifted", found_in_top(random_order, DRIFT_TRAIN), found_in_top(random_order, DRIFT_HELD)])) print(row2(["same model", "drifted", found_in_top(ranking(ENSEMBLE), DRIFT_TRAIN), found_in_top(ranking(ENSEMBLE), DRIFT_HELD)])) print(row2(["model built on drifted data", "drifted", found_in_top(ranking(NEW), DRIFT_TRAIN), found_in_top(ranking(NEW), DRIFT_HELD)])) print(f"\nlabels unchanged: old {positives(HELD_OUT)}, drifted {positives(DRIFT_HELD)} suspicious") print(f"two corrupted features: zero-reading total " f"{sum(a['zero_readings'] for a in HELD_OUT)} -> {sum(a['zero_readings'] for a in DRIFT_HELD)}, " f"subscribers with missing periods {sum(1 for a in HELD_OUT if a['reading_count'] < 3)} -> " f"{sum(1 for a in DRIFT_HELD if a['reading_count'] < 3)}") RETURNS[1][4] = f"{found_in_top(ranking(ENSEMBLE), HELD_OUT)} illicit in top {CAPACITY}" RETURNS.append([STEPS[6], STEPS[1], "measurement policy changed", f"{found_in_top(ranking(ENSEMBLE), DRIFT_HELD)} illicit", f"{found_in_top(ranking(NEW), DRIFT_HELD)} illicit"]) ``` ``` illicit in first 40 visits data train held-out baseline: random order old 10 14 model: nine-tree order old 30 30 baseline: random order drifted 10 14 same model drifted 33 23 model built on drifted data drifted 39 30 labels unchanged: old 93, drifted 93 suspicious two corrupted features: zero-reading total 39 -> 3, subscribers with missing periods 117 -> 0 ``` The same model, the same subscribers, the same labels and a falling number: 23 illicit cases in the first 40 visits instead of 30. Since the baseline of the random order stayed fixed at 14, the drop belongs to the model, it is not the question getting harder. The cause is written in the last line: the two features the model relied on have been erased. The zero-reading total has gone from 39 to 3, the number of subscribers with missing periods from 117 to 0. The model was deciding with the information carried by these two columns, and that information is no longer written to the table. From the seventh step the flow returns to the second. The data is prepared anew according to the new policy, the model is rebuilt on the drifted data and the number returns to 30. The loss is closed, but the price of closing it is a return, and a return is seen only if the monitoring step exists. Without monitoring the list in the field would go on producing seven illicit cases too few and nobody would notice. #### The Accounting of Returns ```python # returns.py — the measure of the flow is not the number of steps but the number of returns. W3 = [16, 24, 30, 16, 22] row3 = lambda h: "".join(str(v).ljust(W3[i]) for i, v in enumerate(h)) print(row3(["from", "to", "reason", "before return", "after return"])) for d in RETURNS: print(row3(d)) print(f"\n{len(STEPS)}-step flow produced {len(RETURNS)} returns; " f"total step visits {sum(int(x[0][0]) - int(x[1][0]) + 1 for x in RETURNS) + len(STEPS)}") ``` ``` from to reason before return after return 5 evaluation 3 feature selection gap below threshold +0.0318 +0.1061 6 deployment 1 problem definition does not fit capacity 240 flagged 30 illicit in top 40 7 monitoring 2 data preparation measurement policy changed 23 illicit 30 illicit 7-step flow produced 3 returns; total step visits 22 ``` All three returns leave from a different step and land on a different step. The return from the fifth step to the third is short and three steps are visited again; the return from the sixth to the first makes six steps be visited again, and the return from the seventh to the second again six steps. The flow, which is seven steps when read linearly, has been completed with 22 step visits, that is, each step has been visited 3.1 times on average. The accounting itself is a decision too. Here a return was counted as the re-execution of all the steps from the target step to the exit step. Some steps may not really need to be repeated — in the return from the sixth to the first, data preparation has not changed — and in that case the number would have come out smaller. Without writing down which accounting produced it, 22 cannot be read. The number of returns is not a measure of defects. A flow with zero returns is a flow whose gates test nothing; had there been no gate in the first pass, a model of 0.0318 would have gone out to the field and nobody would have measured it. A return is the proof that the gate works. #### Summary - The flow is seven steps and is not linear; in this run it produced three returns and was completed with 22 step visits. - The gate at the fifth step tests the gap over the baseline with a threshold of 0.0500: the model with three columns gives 0.0318 and stays at the gate, the model with nine columns gives 0.1061 and passes. - The gate at the sixth step measures not performance but the constraint: the model flags 240 subscribers, the daily capacity is 40, and the constraint turns the problem from classification into ranking; the ranking finds 30 illicit cases in the first 40 visits, the random order 14. - At the seventh step the number falls without the model changing: when the measurement policy changes the same model finds 23 illicit cases, because the two features it relied on have been erased; when rebuilt on the drifted data it returns to 30. - The number of returns is not a measure of defects but proof that the gates work; zero returns means the gates tested nothing. #### Next Step This lesson ran the flow from a single desk: one person wrote the question, chose the features, built the model, set the threshold and monitored the field. In reality these steps are divided among different job definitions, and the division not only shares out the work, it also changes **what is delivered**. The Introduction to Data Analytics course had separated three analyst roles by their acceptance criteria; the next lesson makes the same separation on the machine learning side. The same illicit-use problem turns into three separate outputs in three roles, and the three choose **three separate baselines**: for one role the baseline is the field's current procedure, for another the model already running, for the third the system itself built without a model. ### Roles Source: https://academia.sh/en/courses/introduction-to-machine-learning/roles The same illicit-use problem turns into three separate outputs and three separate baselines in three job definitions. The baseline of the data scientist is the field's current procedure and finds 13 illicit cases in the first 40 visits, the model order delivered finds 30, the gap is 17. The baseline of the machine learning engineer is the model running today and finds 23 on the drifted data, the rebuilt procedure finds 30, the gap is 7. The baseline of the AI engineer is the system without a model and finds 21 in the worst case, the system with a signal finds 30, the gap is 9. The same delivered count produces 17, 7 and 9 in three roles; what determines the number is not the model but the chosen baseline. The system that falls back gives 21 on the drifted data, the one that never falls back 23. The previous lesson ran the flow from a single desk: one person wrote the question, chose the features, built the model, set the threshold and monitored the field. In reality these steps are divided among different job definitions, and the division not only shares out the work, it also changes **what is delivered**. The Introduction to Data Analytics course had separated three analyst roles by their acceptance criteria: the measuring role delivered a number, the role that binds the decision a threshold, the generalizing role a generalization together with its assumption. This lesson makes the same separation on the machine learning side, and what it measures is not the delivery itself but **the baseline each role chooses**. - **FC32.** Role names here are not a title but a **job definition**: what is delivered and by which measure it is accepted. No company, title standard or document is mentioned. - **FC33. The data scientist** delivers a **finding**: whether the problem is predictable and how far. Its baseline is **the field's current procedure**, because what it has to prove is that something better than the present state of the job is possible. - **FC34. The machine learning engineer** delivers a **procedure** that runs every day. Its baseline is **the model running today**, because its job is not to prove something from scratch but to keep something standing. - **FC35. The AI engineer** delivers an end-to-end **system**. Its baseline is **the same system built without a model**, and its measure is not the average but **the worst case**, because a system is as reliable as what it gives on its worst day. - **FC36.** The three roles use the same measure and all three deliver 30, but they produce three separate gain figures. What determines the number is not the model but the chosen baseline. - **FC37.** A fallback path is not counted as an improvement without being measured. In this run the system that falls back finds fewer illicit cases on the drifted data than the one that never falls. #### Rebuilding the Fictional Dataset ```python # setup.py — the fictional dataset of lesson 01 is rebuilt with the same seed: the readings # table, the subscriber summary, the fictional label, the train / held-out split and the tree learner. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] FEATURES = ["mean", "per_person", "drop", "zero_readings", "reading_count", "satisfaction", "household_size", "highest", "lowest"] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 _subscribers, _dropped, _series = [], [], {} for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] k = {"subscriber_id": 10001 + i, "region": b[0], "base": b[2], "household_size": pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]), "satisfaction": pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) + 1} (_dropped if r() < 0.046 else _subscribers).append(k) _valid = {a["subscriber_id"] for a in _subscribers} for k in _subscribers + _dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor(k["base"] * math.exp( (r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] in _valid: _series.setdefault(k["subscriber_id"], []).append(t) r() SUBSCRIBERS = [] for k in _subscribers: t = _series.get(k["subscriber_id"]) if not t: continue mean = sum(t) / len(t) a = {"subscriber_id": k["subscriber_id"], "region": k["region"], "series": t, "household_size": k["household_size"], "satisfaction": k["satisfaction"], "reading_count": len(t), "mean": mean, "highest": max(t), "lowest": min(t), "zero_readings": sum(1 for v in t if v == 0), "per_person": mean / max(1, k["household_size"]), "drop": (t[0] - t[-1]) / t[0] if len(t) > 1 and t[0] > 0 else 0.0} r = rng(SEED + 31000 + a["subscriber_id"]) a["suspicion"] = 1 if (1.30 * a["zero_readings"] + 1.25 * (a["drop"] > 0.45) + 1.05 * (a["per_person"] > 12.0) + 0.65 * (a["reading_count"] < 3) + 0.45 * (a["satisfaction"] <= 2) - 1.55 + (r() + r() - 1.0) * 1.1) > 0 else 0 SUBSCRIBERS.append(a) r = rng(SEED + 90001) TRAIN, HELD_OUT = [], [] for a in SUBSCRIBERS: (TRAIN if r() < 0.70 else HELD_OUT).append(a) positives = lambda k: sum(a["suspicion"] for a in k) accuracy = lambda f, k: sum(1 for a in k if f(a) == a["suspicion"]) / len(k) _impurity = lambda k: 2 * (positives(k) / len(k)) * (1 - positives(k) / len(k)) if k else 0.0 def fit(k, depth=3, min_size=40): """The greedy decision tree of lesson 01: the split criterion and stopping rule are there.""" best, choice = _impurity(k) * len(k), None if depth and len(k) >= min_size: for feat in FEATURES: for threshold in sorted({a[feat] for a in k})[1:]: left = [a for a in k if a[feat] < threshold] right = [a for a in k if a[feat] >= threshold] m = _impurity(left) * len(left) + _impurity(right) * len(right) if m < best - 1e-9: best, choice = m, (feat, threshold, left, right) if choice is None: return 1 if 2 * positives(k) > len(k) else 0 lt, rt = fit(choice[2], depth - 1, min_size), fit(choice[3], depth - 1, min_size) return lt if lt == rt else (choice[0], choice[1], lt, rt) def apply(d, a): while isinstance(d, tuple): d = d[2] if a[d[0]] < d[1] else d[3] return d print(f"fictional dataset (seed {SEED}): {len(SUBSCRIBERS)} subscribers, {len(TRAIN)} train, " f"{len(HELD_OUT)} held-out, {positives(SUBSCRIBERS)} suspicious") ``` ``` fictional dataset (seed 20260218): 1260 subscribers, 883 train, 377 held-out, 327 suspicious ``` #### A Common Measure, Five Rankers For the three roles to be comparable the measure must be common. The common measure comes from the previous lesson: the daily capacity is 40 visits and what is counted is the illicit cases found in the first 40 visits. What is not common is the baseline against which each role reads this measure. The five rankers are measured in two data layouts. The first layout is the table at the beginning of the lesson; the second is the drift built in the previous lesson — the measurement policy has changed, zero readings have been filled with an estimate, missing periods have been completed. The label is the same in both layouts. ```python # rankers.py — the common measure of the three roles: illicit cases found in as many visits as # the daily capacity. Five separate rankers and two separate data layouts are measured side by side. import statistics from collections import Counter CAPACITY = 40 r = rng(SEED + 7205) DEPLOYED = [fit([TRAIN[int(r() * len(TRAIN))] for _ in TRAIN]) for _ in range(9)] r = rng(SEED + 7206) lots = {a["subscriber_id"]: r() for a in SUBSCRIBERS} found_in_top = lambda f, s: positives(sorted(s, key=f)[:CAPACITY]) ranking = lambda T: (lambda a: (-sum(apply(d, a) for d in T), a["subscriber_id"])) field = lambda a: (-a["mean"], a["subscriber_id"]) hand = lambda a: 1 if (a["zero_readings"] >= 1 or a["drop"] > 0.50 or a["per_person"] > 15.0) else 0 rule = lambda a: (-hand(a), -a["mean"], a["subscriber_id"]) def drifted(a): """The measurement policy change of lesson 05: zero readings are filled with an estimate, missing periods are completed. The label does not change.""" filled = [v for v in a["series"] if v > 0] est = round(statistics.fmean(filled), 2) if filled else 0.0 t = [v if v > 0 else est for v in a["series"]] + [est] * (3 - len(a["series"])) avg = sum(t) / len(t) return {**a, "series": t, "reading_count": 3, "mean": avg, "highest": max(t), "lowest": min(t), "zero_readings": sum(1 for v in t if v == 0), "per_person": avg / max(1, a["household_size"]), "drop": (t[0] - t[-1]) / t[0] if t[0] > 0 else 0.0} DRIFT_TRAIN, DRIFT_HELD = [drifted(a) for a in TRAIN], [drifted(a) for a in HELD_OUT] r = rng(SEED + 7305) CURRENT = [fit([DRIFT_TRAIN[int(r() * len(DRIFT_TRAIN))] for _ in DRIFT_TRAIN]) for _ in range(9)] signal = lambda s: sum(1 for a in s if a["zero_readings"] > 0 or a["reading_count"] < 3) / len(s) W = [33, 12, 12, 12, 12] row = lambda h: str(h[0]).ljust(W[0]) + "".join(str(v).rjust(W[i + 1]) for i, v in enumerate(h[1:])) RANKERS = [("random order", lambda a: lots[a["subscriber_id"]]), ("field procedure: top consumer", field), ("hand-written rule order", rule), ("deployed model order", ranking(DEPLOYED)), ("model built on drifted data", ranking(CURRENT))] print(f"illicit cases found in the first {CAPACITY} visits") print(row(["ranker", "old tr.", "old held.", "drift tr.", "drift held."])) COUNTS = {} for name, f in RANKERS: COUNTS[name] = [found_in_top(f, TRAIN), found_in_top(f, HELD_OUT), found_in_top(f, DRIFT_TRAIN), found_in_top(f, DRIFT_HELD)] print(row([name] + COUNTS[name])) print(f"\ndata quality signal (share of subscribers with a zero reading or a missing period): " f"old {signal(HELD_OUT):.4f}, drifted {signal(DRIFT_HELD):.4f}") print(f"the held-out set has {positives(HELD_OUT)} suspicious; the ceiling of {CAPACITY} visits is {CAPACITY}") ``` ``` illicit cases found in the first 40 visits ranker old tr. old held. drift tr. drift held. random order 10 14 10 14 field procedure: top consumer 17 13 18 14 hand-written rule order 25 21 26 21 deployed model order 30 30 33 23 model built on drifted data 40 30 39 30 data quality signal (share of subscribers with a zero reading or a missing period): old 0.3740, drifted 0.0027 the held-out set has 93 suspicious; the ceiling of 40 visits is 40 ``` The table shows five separate procedures in five rows. The random order finds 14 illicit cases and is fixed in both layouts; the difficulty of the question has not changed. The field's current procedure — starting from the highest consumer — finds 13 on the held-out set, that is, one illicit case behind even the random order. The hand-written rule order finds 21. The deployed model finds 30 and falls to 23 on the drifted data. The model rebuilt on the drifted data gives 30 in both layouts. The data quality signal in the last row tells where the drift comes from: the share of subscribers with a zero reading or a missing period has gone from 0.3740 to 0.0027. The two columns the model relied on have been erased from the table, and what measures this is not the model but a counter that looks at the data itself. #### Three Roles, Three Baselines ```python # roles.py — the same model, three job definitions, three separate baselines. Each role chooses its # own baseline and what it delivers is compared with that baseline. WORST = lambda s: min(s[1], s[3]) # the held-out counts of the two data layouts DEPLOYED_NAME = "deployed model order" CURRENT_NAME = "model built on drifted data" FALLBACK = [COUNTS[DEPLOYED_NAME][0], COUNTS[DEPLOYED_NAME][1], COUNTS["hand-written rule order"][2], COUNTS["hand-written rule order"][3]] ROLES = [ ("data scientist", "field's current procedure", COUNTS["field procedure: top consumer"][1], "model order", COUNTS[DEPLOYED_NAME][1]), ("machine learning eng.", "deployed model, on drifted data", COUNTS[DEPLOYED_NAME][3], "rebuilt procedure", COUNTS[CURRENT_NAME][3]), ("AI engineer", "system without a model, worst case", WORST(COUNTS["hand-written rule order"]), "system with signal, worst case", WORST(COUNTS[CURRENT_NAME])), ] W2 = [24, 36, 10, 32, 10, 8] row2 = lambda h: str(h[0]).ljust(W2[0]) + str(h[1]).ljust(W2[1]) + str(h[2]).rjust(W2[2]) \ + " " + str(h[3]).ljust(W2[3]) + str(h[4]).rjust(W2[4]) + str(h[5]).rjust(W2[5]) print(row2(["role", "chosen baseline", "baseline", "delivered", "count", "gap"])) for name, base_name, base, delivered_name, delivered in ROLES: print(row2([name, base_name, base, delivered_name, delivered, f"{delivered - base:+d}"])) print(f"\nfallback system (switches to the rule order when the signal falls): " f"old data {FALLBACK[1]}, drifted data {FALLBACK[3]}, worst {min(FALLBACK[1], FALLBACK[3])}") print(f"system that never falls back: old {COUNTS[DEPLOYED_NAME][1]}, drifted {COUNTS[DEPLOYED_NAME][3]}, " f"worst {WORST(COUNTS[DEPLOYED_NAME])}") ``` ``` role chosen baseline baseline delivered count gap data scientist field's current procedure 13 model order 30 +17 machine learning eng. deployed model, on drifted data 23 rebuilt procedure 30 +7 AI engineer system without a model, worst case 21 system with signal, worst case 30 +9 fallback system (switches to the rule order when the signal falls): old data 30, drifted data 21, worst 21 system that never falls back: old 30, drifted 23, worst 23 ``` The same measure, the same delivered count — and three separate gains: 17, 7 and 9 illicit cases. All three numbers are correct and none of them can be written in place of another. The 17 of the data scientist is a proof: if the field's current procedure finds 13 illicit cases in 40 visits and a model finds 30, this problem is a learnable problem. This number has a short life; once it is proven, nobody writes the same baseline a second time. The 7 of the machine learning engineer is a maintenance number. A model is already running in the field and it gives 23 on the drifted data; the procedure delivered gives 30. Seven illicit cases look small, but the baseline is not 13 but 23 — the job of the engineer is not to produce a gain from zero but not to lose the gain already produced. The 9 of the AI engineer is a resilience number, and it is not a single number but **the worst of two layouts**. The system without a model gives 21 in both layouts; the system with a signal gives 30 in the worst case. The 9 in between is not an average gain but the difference of the lowest guaranteed value. #### Measuring the Fallback Path The last two rows refute a design decision. A system that drops the model and switches to the hand-written rule order when the signal falls finds 21 illicit cases on the drifted data. One that never falls back, that is, keeps working with the corrupted feature, finds 23. Falling back **makes the number two illicit cases worse**. This does not mean the fallback path is unnecessary. The value of the signal lies not in switching to the fallback but in **giving notice**: a counter that falls from 0.3740 to 0.0027 is what triggers the return from the seventh step to the second in the previous lesson, and the model built as a result of that return gives 30. Without the signal the list in the field would go on producing seven illicit cases too few, and in the system that falls back nine illicit cases too few. What the three roles have in common shows here. All three deliver a number, all three write their own baseline next to that number, and the baseline of all three is a choice. When the baseline is not written, all three roles can utter the same sentence: "the system finds 30 illicit cases in 40 visits." This sentence is three times true, three times unmeasured. #### Summary - Role names here are job definitions: what is delivered and by which measure it is accepted. The measure is the same in all three — illicit cases found in the first 40 visits — the baselines differ. - The baseline of the data scientist is the field's current procedure and finds 13 illicit cases; the model order delivered finds 30, the gap is 17. - The baseline of the machine learning engineer is the model running today and finds 23 on the drifted data; the rebuilt procedure delivered finds 30, the gap is 7. - The baseline of the AI engineer is the system without a model and finds 21 in the worst case; the worst case of the system with a signal is 30, the gap is 9. - The system that falls back finds 21 on the drifted data, the one that never falls 23: a fallback path is not counted as an improvement without being measured, and the value of the signal lies not in the fallback but in giving notice. - The same delivered count produces 17, 7 and 9 in three roles; what determines the number is not the model but the chosen baseline. #### Next Step Across six lessons every table had three columns: the baseline, the one measured on training and the one measured on the held-out set. The third column was printed every time and never questioned in any lesson. Why the split is made, at what ratio it is made, how many times the same held-out set may be looked at, why the two numbers diverge was not asked even once. The number was printed, and passed by. Yet in this topic that column tried to speak several times. The hand-written rule passed the baseline on training and stayed below it on the held-out set. The model built with thirty labels gave 0.9400 on training and 0.7745 on the held-out set, and as labels increased the two numbers drew closer. The model built on the drifted data filled all 40 of 40 visits with illicit cases on the training set and stayed at 30 on the held-out set. The same thing happened in three cases: the number measured on the data the model **saw** came out higher than the one measured on the data it **did not see**, and the gap between them was never explained. The next topic takes up this gap. The reason for the split is written, the training, validation and test sets are separated from one another, the point at which the two numbers begin to diverge as the model complexity is raised step by step is counted, how much the same procedure moves across different samples is measured, and four separate ways that make the number measured on the held-out set look larger than it really is are applied one by one. ## Generalization Measuring what a model will do on data it has not seen and how this measure gets corrupted: the reasoning behind separating the training, validation and test sets and how many times the test set may be looked at, the point where the training and held-out numbers diverge as capacity rises, how much the same procedure swings across different training samples, the two numbers moving in opposite directions as the penalty coefficient grows and reading the best point from the validation set, and how much four kinds of leakage inflate the held-out number. ### Training, Validation and Test Sets Source: https://academia.sh/en/courses/introduction-to-machine-learning/training-validation-and-test-sets Measuring that the same model gives two different numbers on data it has seen and data it has not seen: 1,260 subscribers are split 756/252/252, a depth 6 decision tree gives 0.8466 accuracy on training, 0.7897 on validation and 0.7778 on test, and the baseline is also three different numbers on the three sets: 0.7778, 0.8135, 0.7579. As selection is made among 48 candidates produced by the same procedure, the gap between the number on the selection set and the number on the unseen set grows from 0.0211 to 0.0411. On a time-ordered split the test accuracy drops to 0.6865, but the difference above the baseline grows from 0.0198 to 0.0397. Over six lessons, the number the model produced was always measured on the same data. A hand-written rule set and a procedure that extracts its rules from data were put side by side, the number of labels required by four learning types was counted, the difference added by a pretext task was measured, four different baselines were chosen for four problem types, the returns in the workflow were counted and the same problem was shown turning into three different outputs in three roles. In all of them the measure was read from data the model had **seen**. What would happen on data the model had **not seen** was never asked. This lesson asks that question and sets the rule of the course: **performance that has not been measured on a held-out set is not performance, it is memorization.** The question has three faces. How many numbers does the same model give on three sets, how many times can each of these sets be looked at, and by what rule are the sets separated. - **GE1.** The dataset is inherited from the regional measurement network of the Data Analytics curriculum and is **fictional**: a long-format readings table of 3,199 rows, 1,329 subscribers, three periods, five regions, seed 20260218. - **GE2.** The label is added in this curriculum and is **fictional**: a subscriber suspected of illicit use or a fault. It is not a real inspection record; it is produced by applying a threshold to a hidden score. - **GE3.** The modeling unit is the subscriber. Readings are summarized at the subscriber level; 69 subscribers who produced no readings at all are left out, and 1,260 subscribers remain. - **GE4.** The features come from seven fields: region (five binary columns), household size, satisfaction, period count, mean consumption, volatility, zero readings. - **GE5.** The measure is **accuracy**: the proportion of subscribers classified correctly. The classes are imbalanced, and how much accuracy hides this is not the subject of this course. - **GE6.** The baseline is telling every subscriber the most frequent class in the training set. This procedure learns nothing and looks at no feature. - **GE7.** The split ratio is 60/20/20 and it is a decision. The sets are split once and stay the same until the end of the lesson. - **GE8.** The subscriber number is the order of joining the network. The suspicion rate rises along this order; this rise was placed in the setup on purpose and it is what the time-ordered split measures. #### Three Sets, Three Numbers The setup generates the dataset, adds the label, splits it into three and trains a single decision tree. A tree is a model: at each node it chooses a feature and a threshold, splits the set in two and states the majority class at the leaf. Depth 6 is a choice, and its justification is left to the next lesson. ```python # generalization.py — MODEL: the regional measurement network (a fictional setup) of the Data Analytics curriculum is generated with the same # seed; readings are summarized at the subscriber level and a FICTIONAL illicit-use/fault suspicion label # is added on top. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) READINGS = {} for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() DATA = [] for k in SUBSCRIBERS: v = READINGS.get(k["id"]) if v is None: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} for b in REGION: x["reg_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "reg_north", "reg_south", "reg_east", "reg_west", "reg_central"] print(f"readings {sum(len(v) for v in READINGS.values())}, subscribers {len(SUBSCRIBERS)}, " f"labeled subscribers {len(DATA)}, suspicious {sum(x['suspicious'] for x in DATA)}") def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] def split_three(K): a, b = int(len(K) * 0.6), int(len(K) * 0.8) return K[:a], K[a:b], K[b:] TRAIN, VAL, TEST = split_three(shuffle(DATA, SEED + 90000)) print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}") def gini(rows): p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0 return 2 * p * (1 - p) def tree(rows, depth, min_size=2): p = sum(x["suspicious"] for x in rows) / len(rows) best = None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in FIELDS: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = gini(rows) - (len(left) * gini(left) + len(right) * gini(right)) / len(rows) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, min_size), "right": tree(best[4], depth - 1, min_size)} def predict(node, x): while "prediction" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["prediction"] def accuracy(node, rows): return sum(predict(node, x) == x["suspicious"] for x in rows) / len(rows) def baseline(train, rows): c = int(sum(x["suspicious"] for x in train) * 2 > len(train)) return sum(x["suspicious"] == c for x in rows) / len(rows) MODEL = tree(TRAIN, 6) print(f"\n{'procedure':<30} {'train':>8} {'validation':>10} {'test':>8}") print(f"{'baseline (most frequent class)':<30} {baseline(TRAIN, TRAIN):>8.4f} " f"{baseline(TRAIN, VAL):>10.4f} {baseline(TRAIN, TEST):>8.4f}") print(f"{'decision tree, depth 6':<30} {accuracy(MODEL, TRAIN):>8.4f} " f"{accuracy(MODEL, VAL):>10.4f} {accuracy(MODEL, TEST):>8.4f}") ``` ``` readings 3199, subscribers 1329, labeled subscribers 1260, suspicious 276 train 756, validation 252, test 252 procedure train validation test baseline (most frequent class) 0.7778 0.8135 0.7579 decision tree, depth 6 0.8466 0.7897 0.7778 ``` The first row of the table says something unexpected. **The baseline is also three numbers.** The procedure that learns nothing, looks at no feature and gives every subscriber the same answer gives 0.7778 accuracy on the training set, 0.8135 on the validation set and 0.7579 on the test set. The 0.0556 gap between them comes entirely from the randomness of the split: when the suspicious rate moves a few points in a set of 252 subscribers, the accuracy of the most frequent class moves by the same amount. Comparing a model number with the baseline of another set is therefore an error; the baseline is recalculated on every set. The second row is the real number of the lesson. The same model, the same measure, three sets, three numbers — and **two different signs**. On the training set the model beats its own baseline by 6.9 points. On the validation set it falls 2.4 points **below** the baseline. On the test set it beats the baseline by 2.0 points. The 0.8466 read from the training set is not miscalculated; it is calculated correctly but answers the wrong question. That number measures not what the model will do in the future but how much of the past it has memorized. What remains is this question: if one held-out set is enough, why are there two held-out sets. #### How Many Times Can the Test Set Be Looked At The difference between the validation set and the test set lies not in the measure but in the use. The validation set is set aside for **selection**, the test set for **reporting**. The difference between them can be measured. The code below produces 48 candidates from the same procedure: all are depth 6 decision trees, and their only difference is which 85% of the training set they saw. There is no real quality difference among them. ```python POOL = [] for a in range(48): sub = shuffle(TRAIN, SEED + 91000 + a * 17)[:int(len(TRAIN) * 0.85)] m = tree(sub, 6) POOL.append((accuracy(m, VAL), accuracy(m, TEST))) rs = rng(SEED + 92000) print(f"{len(POOL)} candidates, all the same procedure; mean validation " f"{sum(x[0] for x in POOL) / len(POOL):.4f}, best single candidate " f"{max(x[0] for x in POOL):.4f}, mean test " f"{sum(x[1] for x in POOL) / len(POOL):.4f}") print(f"\n{'candidates seen':>15} {'selected validation':>19} " f"{'same model test':>15} {'difference':>10}") for k in (1, 2, 4, 8, 16, 32): d = s = 0.0 for _ in range(100): best = max([POOL[int(rs() * len(POOL))] for _ in range(k)], key=lambda t: t[0]) d, s = d + best[0], s + best[1] print(f"{k:>15} {d / 100:>19.4f} {s / 100:>15.4f} {(d - s) / 100:>+10.4f}") ``` ``` 48 candidates, all the same procedure; mean validation 0.7875, best single candidate 0.8214, mean test 0.7657 candidates seen selected validation same model test difference 1 0.7860 0.7650 +0.0211 2 0.7982 0.7699 +0.0283 4 0.8069 0.7693 +0.0375 8 0.8113 0.7742 +0.0371 16 0.8149 0.7756 +0.0394 32 0.8177 0.7767 +0.0411 ``` What is read is not the **level** of the columns but the **growth** of the difference between them. There is already a constant gap of 0.0218 between validation and test: the mean validation of the forty-eight candidates is 0.7875, the mean test is 0.7657, and this difference stands even when no selection is made. So this gap does not come from selection. What comes from selection is this: as the number of candidates looked at goes from 1 to 32, the validation number of the selected model rises from 0.7860 to 0.8177, that is, by 3.2 points; the test number of **the same models** rises by only 1.2 points. The 2.0 points in between are not something any model gained; they are a number produced by the selection itself. The harshest row is in the first output. The mean validation of the forty-eight candidates is 0.7875, and this is **below** 0.8135, the baseline of the validation set: an average candidate is worse than saying "none is suspicious." Yet the best single candidate reads 0.8214 and appears to have beaten the baseline. A study that tries forty-eight candidates and reports the best one can make a procedure that does not beat the baseline look as though it has. This is the first small example of the second claim of the curriculum, and its full form will be measured in the last lesson of the course. The rule follows from this: the moment a set is looked at and a decision is made according to it, that set turns into a validation set. The test set is looked at once, at the very end, for a single model. A second look makes it a validation set and inflates the reported number by the selection's own share. #### The Split Is a Decision The split so far was random: 1,260 subscribers were shuffled and divided into three. This assumes that every row in the dataset can replace any other. If the network has grown over time and the subscribers who joined later differ from the earlier ones, the assumption does not hold. The code below splits the same data this time **by order of registration**: the first 60% training, the next 20% validation, the last 20% test. ```python TIME_TRAIN, TIME_VAL, TIME_TEST = split_three(sorted(DATA, key=lambda x: x["id"])) TIME_MODEL = tree(TIME_TRAIN, 6) print(f"{'split':<12} {'suspicious rate':>15} {'baseline':>8} {'tree':>8}") for name, s in (("train", TIME_TRAIN), ("validation", TIME_VAL), ("test", TIME_TEST)): print(f"{name:<12} {sum(x['suspicious'] for x in s) / len(s):>15.4f} " f"{baseline(TIME_TRAIN, s):>8.4f} {accuracy(TIME_MODEL, s):>8.4f}") print(f"\nrandom split test: baseline {baseline(TRAIN, TEST):.4f}, tree " f"{accuracy(MODEL, TEST):.4f}, difference " f"{accuracy(MODEL, TEST) - baseline(TRAIN, TEST):+.4f}") print(f"time-ordered split test: baseline {baseline(TIME_TRAIN, TIME_TEST):.4f}, tree " f"{accuracy(TIME_MODEL, TIME_TEST):.4f}, difference " f"{accuracy(TIME_MODEL, TIME_TEST) - baseline(TIME_TRAIN, TIME_TEST):+.4f}") ``` ``` split suspicious rate baseline tree train 0.1561 0.8439 0.9074 validation 0.2738 0.7262 0.7381 test 0.3532 0.6468 0.6865 random split test: baseline 0.7579, tree 0.7778, difference +0.0198 time-ordered split test: baseline 0.6468, tree 0.6865, difference +0.0397 ``` The suspicious rate is 0.1561 on the training set and 0.3532 on the test set. In the random split these three numbers were close to each other, because shuffling distributed the later-joining subscribers equally among the three sets. The time-ordered split gathers them in a single set and sets up the real situation the model will face: learn from the past, predict the future. The result is read in two directions at once, and both are correct. The test accuracy falls from 0.7778 to 0.6865, that is, by 9.1 points — the number given by the random split was too optimistic for a subscriber newly joining the network. But the difference above the baseline **rises** from 0.0198 to 0.0397, that is, it doubles. The reason is that the baseline falls too: the procedure that states the most frequent class of the training set drops to 0.6468 on a test set where the suspicious rate has risen. Someone who looks only at the model number takes the time-ordered split for a decision that "breaks the model." Someone who writes the baseline next to it sees the opposite: the set that got harder penalized the baseline more than the model, and the difference the model actually adds is larger there. The rule of the curriculum is at work in this row — a model is as good as how much it beats a dumber procedure, and that procedure is always remeasured on the same set. #### Summary - The same depth 6 decision tree gives 0.8466 accuracy on the training set, 0.7897 on the validation set and 0.7778 on the test set; since the baseline is also three different numbers on the three sets, 0.7778, 0.8135 and 0.7579, the baseline is recalculated on every set. - The model beats its baseline by 6.9 points on the training set, falls 2.4 points below the baseline on the validation set and beats it by 2.0 points on the test set; a number read from the data it has seen is calculated correctly but answers the wrong question. - Selecting the best among 48 candidates produced by the same procedure raises the validation number by 3.2 points as the candidates looked at go from 1 to 32, while raising the test number by only 1.2 points; the 2.0 points in between are the selection's own share. - The mean validation of the forty-eight candidates is below the baseline at 0.7875, but the best candidate reads 0.8214 and appears to have beaten the baseline; selection can make a procedure that does not beat the baseline look as though it has. - On the time-ordered split the test accuracy falls to 0.6865, but since the baseline also falls to 0.6468 the difference the model adds rises from 0.0198 to 0.0397; the split decision moves both numbers at once. #### Next Step In this lesson depth 6 was chosen without justification and the three numbers were read on top of that choice. Yet depth is the model's **capacity**: it determines how many distinct rules it can learn. When capacity is lowered, both the training number and the held-out number fall; when it is raised, the training number keeps rising but after a point the held-out number turns back. The next lesson raises capacity step by step, prints the two numbers side by side, writes down the point where the divergence begins and names the two deteriorations: underfitting and overfitting. ### Overfitting and Underfitting Source: https://academia.sh/en/courses/introduction-to-machine-learning/overfitting-and-underfitting Measuring how the training and held-out numbers diverge as capacity is raised step by step: as the depth of the decision tree goes from 1 to 14, the training accuracy rises from 0.7778 to 0.9352, the held-out accuracy peaks at 0.8214 and falls to 0.7302, and the difference between the two numbers widens from -0.0357 to +0.2050. Only four of the eleven capacities can match the baseline of 0.8135 on the held-out set. When the training set is raised from 100 to 756 rows, the difference falls from 0.2219 to 0.1680; when the minimum leaf size is raised from 2 to 80, it falls from 0.1680 to -0.0198 and the model turns into exactly the baseline. The previous lesson measured a decision tree on three sets and found three numbers, but it chose the depth of the tree without justification. Depth 6 was not a preference but a gap. Depth is the **capacity** of the tree: it determines how many distinct rules it can learn and into how many pieces it can split the set. Capacity alone is neither good nor bad; it moves two numbers at once, and it moves them in **opposite directions**. This lesson raises capacity from 1 to 14, prints the training and held-out numbers side by side at every step and writes down the point where the two diverge. The two ends of the divergence are two separate deteriorations: **underfitting** when capacity is not enough, **overfitting** when capacity is too much. The test set will not be looked at at all in this lesson; the validation set is used as the held-out set. - **GE9.** The dataset, the label and the split continue unchanged from the previous lesson: 1,260 subscribers, 756 training, 252 validation, 252 test, seed 20260218. The label is fictional. - **GE10.** Capacity is measured by two numbers: the depth of the tree and the number of leaves. The number of leaves gives how many distinct regions the model splits the set into. - **GE11.** The baseline is a tree with a single leaf: the procedure that does not split at all and tells everyone the most frequent class of the training set. It is the zeroth step of the capacity series. - **GE12.** The held-out set is the validation set. The test set is not opened in this lesson; by the rule of the previous lesson it is looked at once, at the close. - **GE13.** The minimum leaf size is a second capacity knob: the smallest number of observations a leaf may fall to. The default is 2. - **GE14.** The learning curve is not drawn, it is printed as numbers: the values the two numbers take while the training set is enlarged make a table. - **GE15.** The held-out set has 252 rows; one subscriber is worth 0.0040 accuracy. Differences below two points are therefore not counted as evidence of a ranking. #### Capacity Rises, the Two Numbers Diverge The setup is the same as in the previous lesson and produces the same dataset with the same seed. A single addition is made on top of it: a leaf counter. ```python # generalization.py — MODEL: the regional measurement network (a fictional setup) of the Data Analytics curriculum is generated with the same # seed; readings are summarized at the subscriber level and a FICTIONAL illicit-use/fault suspicion label # is added on top. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) READINGS = {} for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() DATA = [] for k in SUBSCRIBERS: v = READINGS.get(k["id"]) if v is None: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} for b in REGION: x["reg_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "reg_north", "reg_south", "reg_east", "reg_west", "reg_central"] def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] def gini(rows): p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0 return 2 * p * (1 - p) def tree(rows, depth, min_size=2): p = sum(x["suspicious"] for x in rows) / len(rows) best = None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in FIELDS: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = gini(rows) - (len(left) * gini(left) + len(right) * gini(right)) / len(rows) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, min_size), "right": tree(best[4], depth - 1, min_size)} def predict(node, x): while "prediction" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["prediction"] def accuracy(node, rows): return sum(predict(node, x) == x["suspicious"] for x in rows) / len(rows) def baseline(train, rows): c = int(sum(x["suspicious"] for x in train) * 2 > len(train)) return sum(x["suspicious"] == c for x in rows) / len(rows) def leaves(node): return 1 if "prediction" in node else leaves(node["left"]) + leaves(node["right"]) print(f"{'capacity':>10} {'leaves':>7} {'train':>8} {'held-out':>9} {'difference':>10}") print(f"{'baseline':>10} {1:>7} {baseline(TRAIN, TRAIN):>8.4f} {baseline(TRAIN, VAL):>9.4f} " f"{baseline(TRAIN, TRAIN) - baseline(TRAIN, VAL):>+10.4f}") for dp in (1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14): m = tree(TRAIN, dp) e, h = accuracy(m, TRAIN), accuracy(m, VAL) print(f"{'depth ' + str(dp):>10} {leaves(m):>7} {e:>8.4f} {h:>9.4f} {e - h:>+10.4f}") ``` ``` capacity leaves train held-out difference baseline 1 0.7778 0.8135 -0.0357 depth 1 2 0.7778 0.8135 -0.0357 depth 2 4 0.8016 0.8214 -0.0198 depth 3 8 0.8135 0.8135 +0.0000 depth 4 16 0.8201 0.8175 +0.0026 depth 5 30 0.8360 0.8095 +0.0265 depth 6 46 0.8466 0.7897 +0.0569 depth 7 64 0.8611 0.7698 +0.0913 depth 8 81 0.8770 0.7778 +0.0992 depth 10 112 0.9061 0.7698 +0.1362 depth 12 133 0.9299 0.7619 +0.1680 depth 14 139 0.9352 0.7302 +0.2050 ``` The third column is one-directional: as capacity rises, the training accuracy rises uninterruptedly from 0.7778 to 0.9352. This column is not a measure of success but a measure of capacity — the more splits the model is allowed, the more of the data it has seen it memorizes, and the column rises under every condition. The fourth column is two-directional. The held-out accuracy peaks at 0.8214 at depth 2, then begins to fall and drops to 0.7302 at depth 14. The fifth column is the difference between the two and gives where the divergence begins: the difference crosses zero at depth 3, grows after depth 5 without ever turning back and reaches 0.2050 at depth 14. **The point where the divergence begins lies between depth 4 and 5**: at the fourth step the difference is still 0.0026, while at the fifth it rises to 0.0265 and at the sixth to 0.0569. Read together with the baseline, the table says something harsher. On the held-out set the baseline is 0.8135. Only **four** of the eleven capacities — depth 1, 2, 3 and 4 — reach this number or exceed it; two of them match it exactly, two exceed it and the best beats the baseline by 0.0079. From depth 5 on, **every model is below the baseline**, and what is more, the training column is still rising. At depth 14, the moment the model reads 0.9352 on the training set, it is 8.3 points worse than saying "none is suspicious" on the held-out set. The leaves column shows how the memorization grows. At depth 14 the tree has 139 leaves and the training set has 756 rows: an average of 5.4 subscribers per leaf. A rule built by looking at five subscribers classifies those five subscribers correctly and says nothing about anything else. #### The Rule the Tree Builds Where the divergence comes from becomes visible when one looks at what the model has learned. The code below writes out the rules of the tree at the top explicitly and counts how many subscribers the leaves of the tree at the bottom looked at. ```python def rules(node, rows, path=""): if "prediction" in node: print(f" {path or '(no split)':<50} {len(rows):>4} subscribers -> {node['prediction']}") return f, v = node["field"], node["threshold"] rules(node["left"], [x for x in rows if x[f] < v], f"{path}{f} < {v} ") rules(node["right"], [x for x in rows if x[f] >= v], f"{path}{f} >= {v} ") print("rules of the depth 2 tree") rules(tree(TRAIN, 2), TRAIN) def sizes(node, rows, c): if "prediction" in node: c.append(len(rows)) return f, v = node["field"], node["threshold"] sizes(node["left"], [x for x in rows if x[f] < v], c) sizes(node["right"], [x for x in rows if x[f] >= v], c) c = [] sizes(tree(TRAIN, 14), TRAIN, c) c.sort() print(f"\ndepth 14: {len(c)} leaves, smallest {c[0]}, median {c[len(c) // 2]}, " f"largest {c[-1]} subscribers") print(f"leaves that set a rule after seeing fewer than 5 subscribers: {sum(1 for n in c if n < 5)}") print(f"training subscribers covered by these leaves: {sum(n for n in c if n < 5)}") ``` ``` rules of the depth 2 tree volatility < 11.69 mean_consumption < 18.54 262 subscribers -> 0 volatility < 11.69 mean_consumption >= 18.54 249 subscribers -> 0 volatility >= 11.69 reg_east < 1 185 subscribers -> 0 volatility >= 11.69 reg_east >= 1 60 subscribers -> 1 depth 14: 139 leaves, smallest 2, median 3, largest 141 subscribers leaves that set a rule after seeing fewer than 5 subscribers: 112 training subscribers covered by these leaves: 300 ``` **Three of the four leaves of the tree at the top say the same thing as the baseline.** The only place where the model departs from the baseline is the 60-row leaf made up of east subscribers whose volatility is above 11.69 m³. The whole of the 0.0079 gain on the held-out set comes from this single rule. A model beating the baseline is often not a matter of beating the baseline everywhere but of finding the narrow region where the baseline is wrong. The tree at the bottom is the opposite. The median of the hundred and thirty-nine leaves looks at **three subscribers**, 112 leaves have seen fewer than five subscribers, and these leaves cover 300 rows of the training set, that is, 40%. A large part of the 0.9352 in the training column consists of knowing those three subscribers correctly with rules drawn from three subscribers. On the held-out set these rules have no counterpart. #### The Two Deteriorations Have Separate Signatures The two ends of the table are two separate deteriorations and their symptoms do not mix. **Underfitting** is in the upper rows. The depth 1 tree builds two leaves, but both leaves say the same class: the tree has split, the model has not. Its numbers are exactly the same as the baseline (0.7778 and 0.8135). Its signature is this: **the training number sticks to its own baseline.** If the model cannot explain even the data it has seen, it cannot explain the data it has not seen at all, and to understand this one does not even need to look at the held-out set. **Overfitting** is in the lower rows. Its signature requires reading two numbers together: a high training number, a low held-out number, a large difference. On its own, 0.9352 says nothing — it is neither good nor bad, because it cannot be interpreted unless it is written with which capacity produced it. The sign of the difference is not enough on its own either. At depth 1 the difference looks "safe" at -0.0357, yet that row is the row where the model has learned nothing. A negative difference is not good news; it is often news of underfitting. Three numbers are read together: baseline, training, held-out. #### The Measure of the Correction There are two known corrections for overfitting: enlarging the training set and restricting capacity. Both can be measured. The code below fixes the depth at 12 and changes first the training set, then the minimum leaf size. ```python print("training set grows, capacity fixed (depth 12)") print(f"{'train rows':>13} {'leaves':>7} {'train':>8} {'held-out':>9} {'difference':>10}") for n in (100, 200, 400, 756): m = tree(TRAIN[:n], 12) e, h = accuracy(m, TRAIN[:n]), accuracy(m, VAL) print(f"{n:>13} {leaves(m):>7} {e:>8.4f} {h:>9.4f} {e - h:>+10.4f}") print("\ncapacity is restricted, training set fixed (depth 12, leaf minimum)") print(f"{'leaf min_size':>13} {'leaves':>7} {'train':>8} {'held-out':>9} {'difference':>10}") for z in (2, 5, 10, 20, 40, 80): m = tree(TRAIN, 12, z) e, h = accuracy(m, TRAIN), accuracy(m, VAL) print(f"{z:>13} {leaves(m):>7} {e:>8.4f} {h:>9.4f} {e - h:>+10.4f}") ``` ``` training set grows, capacity fixed (depth 12) train rows leaves train held-out difference 100 15 0.9600 0.7381 +0.2219 200 34 0.9500 0.7500 +0.2000 400 68 0.9450 0.7659 +0.1791 756 133 0.9299 0.7619 +0.1680 capacity is restricted, training set fixed (depth 12, leaf minimum) leaf min_size leaves train held-out difference 2 133 0.9299 0.7619 +0.1680 5 81 0.8624 0.7540 +0.1085 10 44 0.8307 0.7937 +0.0370 20 23 0.8148 0.7500 +0.0648 40 13 0.8042 0.7897 +0.0146 80 7 0.7937 0.8135 -0.0198 ``` The first table is the learning curve, and its direction is right, its strength limited. When the training set is raised from 100 to 756 rows, that is, by a factor of 7.6, the difference falls from 0.2219 to 0.1680: only a quarter of the initial gap has closed. The held-out number rises from 0.7381 to 0.7619 but is still 5.2 points below the baseline of 0.8135. **Adding data pays part of the debt of an overchosen capacity, not all of it.** The second reading of the same curve is the training column: it falls from 0.9600 to 0.9299, because memorizing more rows with the same capacity is harder. The second table touches capacity directly, and its two ends are clear. When the minimum leaf size is raised from 2 to 80, the number of leaves falls from 133 to 7 and the difference from 0.1680 to -0.0198. The last row is striking: when capacity is restricted enough, the model reads exactly 0.8135 on the held-out set, that is, it **turns into the baseline itself**. Restricting capacity to correct overfitting, when carried far enough, ends in underfitting. The two deteriorations bound the usable region from both sides. The rows in between are read with care. The held-out column falls from 0.7937 to 0.7500 and returns to 0.7897; these are swings of 11 and 10 subscribers and stay within the margin of variation measured in the Statistical Analysis and Exploratory Data Analysis course. To count the ranking in the middle of this column as evidence of capacity would be, once again, to take a selected number for performance. The reliable reading is at the two ends and in the difference column. #### Summary - As capacity rises the training accuracy rises in one direction (from 0.7778 to 0.9352), the held-out accuracy peaks at 0.8214 at depth 2 and falls to 0.7302; the difference between the two numbers widens from -0.0357 to +0.2050 and the divergence begins between depth 4 and 5. - Of the eleven capacities only four match the baseline of 0.8135 on the held-out set; after depth 5 every model is below the baseline, and what is more, the training column keeps rising. - The signature of underfitting is the training number sticking to its own baseline and it is seen without looking at the held-out set; the signature of overfitting becomes visible only when two numbers are read together. - Enlarging the training set by a factor of 7.6 lowers the difference from 0.2219 to 0.1680, that is, closes a quarter of the gap; adding data does not pay the whole debt of an overchosen capacity. - Raising the minimum leaf size from 2 to 80 lowers the number of leaves from 133 to 7 and the difference to -0.0198, and the model turns into exactly the baseline on the held-out set: the correction of overfitting, when it goes far enough, ends in underfitting. #### Next Step In this lesson the difference was measured on a single training set. Yet that set of 756 rows is **one** sample drawn from among 1,260 subscribers; on another draw another tree would have been built. This is what lies beneath overfitting: the model produced by a high-capacity procedure swings a lot when the training sample changes; a low-capacity procedure always stands in the same place but is always wrong in the same place. The next lesson splits the error into these two components — systematic deviation and the swing from sample to sample — and counts each separately by building multiple training sets with sampling with replacement. ### Bias–Variance Tradeoff Source: https://academia.sh/en/courses/introduction-to-machine-learning/bias-variability-balance Splitting the error on the held-out set into two components: on training sets built by sampling 30 times with replacement, the total squared error comes to 41.242 for the baseline, 29.768 for the depth 3 tree and 33.810 for the depth 12 tree, in m³ squared; the variance component rises from 0.040 to 4.409 while the bias and noise component falls from 41.202 to 28.448 and stops there. The bias that capacity can remove is 12.754 m³ squared, and the remaining 28.4 m³ squared cannot be removed by any capacity. For the same subscriber, the mean range of the predictions given by 30 models is 3.587 m³ at depth 1 and 8.172 m³ at depth 12. The previous lesson raised capacity and showed two numbers diverging, but the entire measurement was made on **a single** training set. Yet those 756 rows are one sample drawn from among 1,260 subscribers. On another draw another tree is built, other thresholds are chosen and a different answer is produced for the same subscriber. Part of the error on the held-out set comes precisely from here: not from the model being **wrong**, but from the model **swinging**. This lesson splits the error into two components. The first is how much the same procedure swings as the training sample changes: **variance**. The second is how far the mean prediction, taken over all the samples, deviates from the true value: **bias**. Both are tied to the same knob, capacity, in opposite directions, and their sum is smallest in the middle. - **GE16.** In this lesson the problem is **regression**: estimating a numeric value. In the Statistical Analysis and Exploratory Data Analysis course of the Data Analytics curriculum, "regression" was used to interpret the coefficient of a relationship; here the measure is not a coefficient but the error of the number produced for a subscriber that has not been seen. - **GE17.** The target is the subscriber's mean consumption (m³) and is derived from the readings. The features contain no field derived from the readings: region, household size, satisfaction, number of periods. - **GE18.** The measure is the root mean squared error (m³). The decomposition is made in squared units, because additivity holds only there. - **GE19.** The set, the split and the seed continue unchanged from the previous lessons: 1,260 subscribers, 756 training, 252 held-out. The test set is again not opened. - **GE20.** Sampling with replacement was set up in the same course of the Data Analytics curriculum and the method is not repeated here. Its only use is to measure as if there were thirty training sets while only one is at hand. - **GE21.** For a fixed subscriber and $B$ models the following equality is exact: the mean squared error is the sum of the scatter of the predictions among themselves and the square of the distance of the mean prediction from the true value. The three columns in the table are this equality. - **GE22.** The third column carries the bias **and the irreducible noise together**; with a single observation the two cannot be separated. Because the noise is independent of capacity, the **differences** in that column are differences in bias. #### The Same Two Numbers in Regression The setup generates the same regional measurement network (a fictional setup) with the same seed, takes the target as the subscriber's mean consumption and builds a regression tree. The tree has the same structure as the classification tree; the only differences are that the leaf says the **mean** instead of the majority class and that the split is chosen not by class purity but by squared deviation. ```python # regression.py — MODEL. The regional measurement network (a fictional setup) of the Data Analytics curriculum is generated with the same seed; # the target is the subscriber's mean consumption and is derived from the readings. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) DATA = [] for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "target": round(sum(v) / len(v), 2)} for b in REGION: x["reg_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) FIELDS = ["household", "satisfaction", "period", "reg_north", "reg_south", "reg_east", "reg_west", "reg_central"] def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, HELD = K[:756], K[756:1008] def mean_sq_dev(s): m = sum(x["target"] for x in s) / len(s) return sum((x["target"] - m) ** 2 for x in s) / len(s) def reg_tree(s, depth, min_size=5): best = None if depth and len(s) >= 2 * min_size: for f in FIELDS: values = sorted({x[f] for x in s}) for v in (values[1:] if len(values) < 9 else [values[int(i * len(values) / 9)] for i in range(1, 9)]): left = [x for x in s if x[f] < v] right = [x for x in s if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = (mean_sq_dev(s) - (len(left) * mean_sq_dev(left) + len(right) * mean_sq_dev(right)) / len(s)) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": sum(x["target"] for x in s) / len(s)} return {"field": best[1], "threshold": best[2], "left": reg_tree(best[3], depth - 1, min_size), "right": reg_tree(best[4], depth - 1, min_size)} def predict(d, x): while "prediction" not in d: d = d["left"] if x[d["field"]] < d["threshold"] else d["right"] return d["prediction"] def rmse(f, s): return math.sqrt(sum((f(x) - x["target"]) ** 2 for x in s) / len(s)) MEAN = sum(x["target"] for x in TRAIN) / len(TRAIN) print(f"train {len(TRAIN)}, held-out {len(HELD)}, train mean {MEAN:.3f} m3") print(f"\n{'procedure':<14} {'train':>9} {'held-out':>9}") print(f"{'baseline':<14} {rmse(lambda x: MEAN, TRAIN):>9.3f} {rmse(lambda x: MEAN, HELD):>9.3f}") for dp in (1, 3, 12): m = reg_tree(TRAIN, dp) print(f"{'depth ' + str(dp):<14} {rmse(lambda x: predict(m, x), TRAIN):>9.3f} " f"{rmse(lambda x: predict(m, x), HELD):>9.3f}") ``` ``` train 756, held-out 252, train mean 19.767 m3 procedure train held-out baseline 6.200 6.419 depth 1 5.696 5.979 depth 3 5.105 5.380 depth 12 4.639 5.594 ``` The baseline here is "saying the mean": every subscriber is assigned 19.767 m³, the mean of the training set. It gives an error of 6.419 m³ on the held-out set. The depth 3 tree lowers this to 5.380 m³. The depth 12 tree is best on the training set with 4.639 m³ but rises to 5.594 m³ on the held-out set — the shape of the previous lesson, this time in cubic meters. The table does not say what has gone wrong. Is the depth 12 tree aiming at a wrong place, or does it aim at the right place and land somewhere different every time? The corrections for the two cases are different, and the two must be separated. #### Splitting the Error in Two The way to separate them is to multiply the single training set. Sampling with replacement builds 30 different sets of 756 rows from the same training set; on each of them the same procedure is run and 30 separate answers are produced for the subscribers in the same held-out set. ```python B = 30 SAMPLES = [] for b in range(B): r = rng(SEED + 61000 + b * 13) SAMPLES.append([TRAIN[int(r() * len(TRAIN))] for _ in range(len(TRAIN))]) def decompose(name, fit, predictor): MODELS = [fit(s) for s in SAMPLES] PRED = [[predictor(m, x) for x in HELD] for m in MODELS] total = variance = bias = 0.0 for i, x in enumerate(HELD): p = [PRED[b][i] for b in range(B)] m = sum(p) / B total += sum((q - x["target"]) ** 2 for q in p) / B variance += sum((q - m) ** 2 for q in p) / B bias += (m - x["target"]) ** 2 n = len(HELD) print(f"{name:<14} {total / n:>9.3f} {variance / n:>9.3f} {bias / n:>9.3f} " f"{math.sqrt(total / n):>7.3f}") print(f"{B} samples drawn with replacement, {len(TRAIN)} rows each") print(f"\n{'procedure':<14} {'total':>9} {'variance':>9} {'bias':>9} {'rmse':>7}") decompose("baseline", lambda s: sum(y["target"] for y in s) / len(s), lambda m, x: m) for dp in (1, 2, 3, 5, 8, 12): decompose(f"depth {dp}", lambda s, dp=dp: reg_tree(s, dp), predict) ``` ``` 30 samples drawn with replacement, 756 rows each procedure total variance bias rmse baseline 41.242 0.040 41.202 6.422 depth 1 36.158 2.138 34.020 6.013 depth 2 31.840 2.218 29.622 5.643 depth 3 29.768 1.321 28.448 5.456 depth 5 30.745 2.274 28.471 5.545 depth 8 32.922 3.983 28.939 5.738 depth 12 33.810 4.409 29.400 5.815 ``` The second and third columns give the first exactly in every row; the equality is not an approximation but the definition itself. The columns move in opposite directions. The variance rises from 0.040 to 4.409, that is, by more than a hundredfold. The bias and noise fall from 41.202 to 28.448 and stop there: after depth 3 this column does not fall, it rises slightly. The baseline is one end of the tradeoff and its place in the table shows it. Its variance is almost zero at 0.040 — whichever sample it is trained on, it says almost the same number. The price of this is a bias of 41.202: it always stands in the same place and is always wrong in the same place. Stability is not a virtue on its own. The **floor** of the third column is the most important number of the lesson. Whatever is done with capacity, this column does not go below 28.4 m³ squared; this is the remaining noise that no feature can explain. The bias that capacity actually removes is 12.754 m³ squared between 41.202 and 28.448, that is, 30.9% of the error of the baseline. The remaining 69.1% is removed neither by a deeper tree nor by more data; it is the measurement network's own scatter. The third column has a second reading, and it gives the rationale of a family of methods. That column is the error on the held-out set of a procedure that takes the **mean** of the thirty models: when a mean is taken, by definition the variance component vanishes and only the third column remains. For depth 12, a single tree makes a squared error of 33.810 m³ while the mean of thirty trees makes 29.400, that is, 13.0% of the error is erased merely by averaging; on the root scale, 5.422 m³ instead of 5.815 m³. This number is smaller even than the 5.456 m³ of a single depth 3 tree. **Ensemble methods** do exactly this: instead of trying to lower the bias, they multiply high-capacity, swinging models and leave the variance to the average. The way to lower the error without touching the capacity knob at all passes through here. The variance column is not a one-directional function of capacity, and the reason can be read. At depth 3 the variance, 1.321, is below the 2.218 of depth 2. At depth 1 and 2 the tree has to separate five regions with one or two splits, and **which region it will separate** changes from sample to sample; at depth 3 the five regions get their own leaf in almost every sample and the thirty models resemble each other. When capacity is insufficient, the swinging comes not from what the model has learned but from **what it has been forced to choose**. #### The Same Subscriber, Thirty Models The components are averages; what they look like in a single subscriber is asked separately. ```python print(f"{'capacity':<12} {'mean range':>17} {'widest subscriber':>17}") MODELS = {} for dp in (1, 3, 12): MODELS[dp] = [reg_tree(s, dp) for s in SAMPLES] a = [max(t) - min(t) for t in ([predict(m, x) for m in MODELS[dp]] for x in HELD)] print(f"{'depth ' + str(dp):<12} {sum(a) / len(a):>17.3f} {max(a):>17.3f}") print(f"\n{'subscriber':>10} {'region':<8} {'actual':>7}", end="") for dp in (1, 3, 12): print(f" {'d' + str(dp) + ' mean':>12} {'range':>7}", end="") print() for x in (HELD[3], HELD[11], HELD[40]): b = [k for k in ("north", "south", "east", "west", "central") if x["reg_" + k]][0] print(f"{x['id']:>10} {b:<8} {x['target']:>7.2f}", end="") for dp in (1, 3, 12): p = [predict(m, x) for m in MODELS[dp]] print(f" {sum(p) / len(p):>12.2f} {max(p) - min(p):>7.2f}", end="") print() ``` ``` capacity mean range widest subscriber depth 1 3.587 5.246 depth 3 4.595 16.362 depth 12 8.172 24.236 subscriber region actual d1 mean range d3 mean range d12 mean range 10227 north 18.92 20.25 2.87 20.52 2.92 21.36 6.54 10683 west 11.99 15.66 5.25 13.80 5.16 13.72 5.48 10424 south 26.22 20.25 2.87 16.94 4.05 16.15 9.17 ``` In the first row the bias stands bare. At depth 1, subscriber 10227 from the north and subscriber 10424 from the south get **the same number**: 20.25 m³. Although the true values of the two subscribers are 18.92 and 26.22 m³, the model cannot tell them apart, because its single split leaves these two regions in the same leaf. Its range is narrow at 2.87 — it is stably wrong. At depth 3 the prediction for 10424 drops to 16.94 m³, and this is **not a worse but a more accurate** number: it is close to the true mean of the south region. That the subscriber's own value is 26.22 m³ is not a sign of bias but the very thing that noise is — that subscriber's readings came out above those of its region, and no feature in the set announces this. To expect a model to know this is to expect it to remove the floor of the third column. At depth 12 the mean prediction stays almost the same at 16.15, but the range rises from 4.05 to 9.17 m³. The same thing happens over the whole held-out set: the mean range is 3.587 at depth 1 and 8.172 m³ at depth 12. In the widest subscriber the answers of the thirty models spread over a range of 24.236 m³, and this is larger than the target's own mean of 19.767 m³. The answer given by high capacity is not wrong, it is **random**; it depends on which training sample was drawn. #### Summary - The mean squared error on the held-out set is the exact sum of two components: the scatter of the predictions from sample to sample and the distance of the mean prediction from the true value. - As capacity rises the variance goes from 0.040 up to 4.409 and the bias and noise from 41.202 down to 28.448; the total is smallest at depth 3 with 29.768 and its root measure is 5.456 m³. - The floor of the third column is 28.4 m³ squared and is irreducible noise; the bias removed by capacity is 12.754 m³ squared, that is, 30.9% of the baseline's error. The same column is the error of an ensemble method that takes the mean of thirty models and brings 33.810 down to 29.400 at depth 12. - The baseline is one end of the tradeoff: its variance is almost zero at 0.040 and its bias is the largest at 41.202; stability alone is not a virtue. - The variance is not a one-directional function of capacity; at depth 1 and 2 the swinging is greater than at depth 3, because which region the tree will separate changes from sample to sample. - At the level of a single subscriber, depth 1 gives the subscribers of two different regions the same number, 20.25 m³, while depth 12 produces a range of 9.17 m³ among thirty models. #### Next Step The tradeoff has been found but the way it was found is crude: depth is an integer and there is no intermediate value between 3 and 5. Instead of cutting capacity off, **penalizing** it gives a finer knob — the complexity of the model is added to the error measure as a cost and the penalty coefficient is tuned continuously. The next lesson raises the penalty coefficient step by step, prints that the training number gets worse without interruption while the held-out number first improves and then gets worse, and shows from which set the best point is read. ### Regularization Intuition Source: https://academia.sh/en/courses/introduction-to-machine-learning/regularization-intuition Measuring how penalizing complexity moves two numbers in opposite directions: in a linear model with 29 columns on 120 rows, when the penalty coefficient is raised from 0.001 to 1000 the weight magnitude falls from 10.040 to 0.601 and the training error worsens from 4.603 to 5.453 m³, while the validation error falls from 5.924 to 5.553 and returns to 5.932. Decomposed by sampling with replacement, the variance falls from 12.988 to 0.181, while the bias hits a floor at 30.976 and jumps to 35.295. The best penalty read from validation is 100 and gives 5.559 m³ on the test set; the baseline is 6.556 and the unpenalized model 5.841 m³. The previous two lessons restricted capacity with a crude knob: the depth of the tree is an integer, there is no intermediate value between 3 and 5, and the same goes for the leaf lower bound. Yet the complexity of a model can be **penalized** instead of being cut off. Next to the error measure, a term that measures how assertive the model is is added, and the model is forced into a compromise between lowering the error and lowering its assertiveness. The penalty coefficient is a continuous number; between two integers there are as many stops as one likes. This lesson raises that coefficient from 0.001 to 1000 and prints three things at every step: the magnitude of the weights, the training error and the held-out error. The expected behavior is not one-directional — the training error worsens without interruption, the held-out error first improves and then worsens — and from which set the best point is to be read is the last question of the lesson. - **GE23.** The model is linear and has 29 columns: five region indicators, household size, satisfaction, number of periods, their squares, their pairwise products and the product of each with the region. - **GE24.** The penalty is a quadratic term: $\lambda$ times the sum of squares of the weights is added to the error measure. Instead of a constant term, the training mean is subtracted from the target, so that all the weights are penalized and, as the penalty grows, the model converges to the **baseline**. - **GE25.** The columns are scaled by the mean and the scatter of the training set. The penalty is sensitive to scale and **from which set** the scale is computed is a decision. - **GE26.** The training set is deliberately cut down to 120 rows. 120 rows for twenty-nine columns is the ratio that makes overfitting measurable. - **GE27.** The validation and test sets are the 252-row sets of the previous lessons; the seed and the split are the same. - **GE28.** The measure is the root mean squared error (m³); the decomposition is made in squared units. - **GE29.** The baseline is the model in which all the weights are zero: telling every subscriber the mean of the training set. - **GE30.** The test set is looked at once in this lesson, after the selection is finished. #### The Penalty Moves the Two Numbers in Opposite Directions The setup generates the set with the same seed, builds the linear model and raises the penalty coefficient in eleven steps. The solution is in closed form: the weights that minimize the squared error are the solution of a system of equations with the penalty added to its diagonal. ```python # regularization.py — MODEL. The same regional measurement network (a fictional setup), the same seed. The target is # the subscriber's mean consumption; the model is linear, the penalty quadratic. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) DATA = [] for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if v: DATA.append({"id": k["id"], "region": k["region"][0], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "target": round(sum(v) / len(v), 2)}) def columns(x): t = [float(x["region"] == b[0]) for b in REGION] h, m, d = float(x["household"]), float(x["satisfaction"]), float(x["period"]) return (t + [h, m, d, h * h, m * m, d * d, h * m, h * d, m * d] + [g * h for g in t] + [g * m for g in t] + [g * d for g in t]) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:120], K[756:1008], K[1008:] P = len(columns(TRAIN[0])) RAW_TRAIN = [columns(x) for x in TRAIN] MEANS = [sum(s[j] for s in RAW_TRAIN) / len(RAW_TRAIN) for j in range(P)] SPREADS = [max(1e-9, math.sqrt(sum((s[j] - MEANS[j]) ** 2 for s in RAW_TRAIN) / len(RAW_TRAIN))) for j in range(P)] MEAN = sum(x["target"] for x in TRAIN) / len(TRAIN) def scale(x): s = columns(x) return [(s[j] - MEANS[j]) / SPREADS[j] for j in range(P)] def solve(A, b): n = len(b) for i in range(n): p = max(range(i, n), key=lambda k: abs(A[k][i])) A[i], A[p], b[i], b[p] = A[p], A[i], b[p], b[i] for k in range(i + 1, n): f = A[k][i] / A[i][i] for j in range(i, n): A[k][j] -= f * A[i][j] b[k] -= f * b[i] w = [0.0] * n for i in range(n - 1, -1, -1): w[i] = (b[i] - sum(A[i][j] * w[j] for j in range(i + 1, n))) / A[i][i] return w def fit(rows, penalty): X = [scale(x) for x in rows] y = [x["target"] - MEAN for x in rows] A = [[sum(X[i][a] * X[i][b] for i in range(len(X))) + (penalty if a == b else 0.0) for b in range(P)] for a in range(P)] return solve(A, [sum(X[i][a] * y[i] for i in range(len(X))) for a in range(P)]) def rmse(w, s): return math.sqrt(sum((MEAN + sum(a * b for a, b in zip(w, scale(x))) - x["target"]) ** 2 for x in s) / len(s)) print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}, columns {P}") print(f"baseline: train {rmse([0.0] * P, TRAIN):.3f} validation {rmse([0.0] * P, VAL):.3f} " f"test {rmse([0.0] * P, TEST):.3f}") print(f"\n{'penalty':>8} {'weight':>9} {'train':>9} {'validation':>10} {'test':>8}") for c in (0.001, 0.01, 0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000): w = fit(TRAIN, c) print(f"{c:>8} {math.sqrt(sum(v * v for v in w)):>9.3f} {rmse(w, TRAIN):>9.3f} " f"{rmse(w, VAL):>10.3f} {rmse(w, TEST):>8.3f}") ``` ``` train 120, validation 252, test 252, columns 29 baseline: train 6.121 validation 6.439 test 6.556 penalty weight train validation test 0.001 10.040 4.603 5.924 5.841 0.01 9.963 4.603 5.921 5.839 0.1 9.334 4.604 5.894 5.815 0.3 8.396 4.606 5.850 5.777 1 6.687 4.620 5.770 5.704 3 4.891 4.653 5.693 5.634 10 3.291 4.716 5.614 5.571 30 2.222 4.804 5.556 5.534 100 1.534 4.920 5.553 5.559 300 1.099 5.089 5.657 5.694 1000 0.601 5.453 5.932 6.008 ``` The weight column shows directly what the penalty does: the magnitude of the weights falls from 10.040 to 0.601, that is, to one seventeenth. The penalty does nothing but shrink the weights of the model; it does not touch the columns, it does not drop rows, it does not delete rules. The third column is one-directional and it is **worsening**: the training error rises from 4.603 to 5.453 m³. This is expected, because the unpenalized solution is already the solution that makes the squared error on the training set smallest; every penalty put on top of it can only spoil that number. The worsening of the training error is not a side effect of regularization but the proof that it is working. The fourth column is two-directional: it falls from 5.924 to 5.553, then returns to 5.932. The lowest point is at penalty 100 and gives a gain of 0.371 m³ over the unpenalized model. The distance between the fall and the rise is not narrow: every value of the penalty between 3 and 300 is better than the unpenalized model, but at 1000 the model is worse than even the unpenalized model. The direction of the last row is the intuition of the lesson. As the penalty grows the weights approach zero, and when all the weights are zero the model tells every subscriber the training mean — that is, it turns into **the baseline itself**. Regularization is a force that pulls the model toward the baseline with a continuous knob. At penalty 1000 the weight, 0.601, is still not zero, and the validation error, 5.932, is below the baseline's 6.439; that knob has not yet been turned all the way. Looking at the weights themselves shows what the penalty drives out. ```python NAMES = ([b[0] for b in REGION] + ["household", "satisfaction", "period", "household^2", "satisfaction^2", "period^2", "household*satisfaction", "household*period", "satisfaction*period"] + [b[0] + "*household" for b in REGION] + [b[0] + "*satisfaction" for b in REGION] + [b[0] + "*period" for b in REGION]) for c in (0.001, 100): w = fit(TRAIN, c) top = sorted(range(P), key=lambda j: -abs(w[j]))[:4] print(f"penalty {c}: four largest weights") for j in top: print(f" {NAMES[j]:<20} {w[j]:>8.3f}") t = sum(abs(w[j]) for j in range(P)) print(f" share of the top four weights in the total weight: " f"{sum(abs(w[j]) for j in top) / t:.3f}") ``` ``` penalty 0.001: four largest weights south -4.138 period 2.981 household^2 2.965 household -2.962 share of the top four weights in the total weight: 0.303 penalty 100: four largest weights west*period -0.574 east*period 0.553 west -0.487 east 0.479 share of the top four weights in the total weight: 0.302 ``` The weights of `household` and `household^2` in the unpenalized solution are -2.962 and +2.965: they almost cancel each other out. Because the two columns are two forms of the same information, the model can give both large and opposite weights; the predictions do not change, the weights swell. The penalty forbids exactly this. At penalty 100 the four largest weights are the west and east regions and the terms of these tied to the number of periods — this is the real structure in the fiction, because the region determines the base consumption. The last line says **how** the penalty works. The share of the first four weights in the total weight is the same at both penalties: 0.303 and 0.302. The quadratic penalty does not eliminate columns, it does not zero out any of them, it does not change the **shape** of the distribution; it shrinks them all together. This is the difference from restricting capacity — tree pruning deletes a rule, the penalty softens all the rules by the same proportion. #### Which Component of the Error the Penalty Shrinks The decomposition of the previous lesson can be applied as is. If the penalty is a knob that restricts capacity, it must lower the variance component of the error and raise the bias component. ```python B = 30 SAMPLES = [] for b in range(B): r = rng(SEED + 61000 + b * 13) SAMPLES.append([TRAIN[int(r() * len(TRAIN))] for _ in range(len(TRAIN))]) print(f"{B} samples drawn with replacement, {len(TRAIN)} rows each") print(f"\n{'penalty':>8} {'total':>9} {'variance':>9} {'bias':>9}") for c in (0.001, 1, 30, 100, 1000): PRED = [] for s in SAMPLES: w = fit(s, c) PRED.append([MEAN + sum(a * b for a, b in zip(w, scale(x))) for x in VAL]) total = variance = bias = 0.0 for i, x in enumerate(VAL): p = [PRED[b][i] for b in range(B)] m = sum(p) / B total += sum((q - x["target"]) ** 2 for q in p) / B variance += sum((q - m) ** 2 for q in p) / B bias += (m - x["target"]) ** 2 n = len(VAL) print(f"{c:>8} {total / n:>9.3f} {variance / n:>9.3f} {bias / n:>9.3f}") ``` ``` 30 samples drawn with replacement, 120 rows each penalty total variance bias 0.001 47.027 12.988 34.039 1 38.330 5.133 33.196 30 32.689 1.622 31.066 100 32.075 1.098 30.976 1000 35.476 0.181 35.295 ``` The variance column is the real target of the penalty: it falls from 12.988 to 0.181, that is, to one seventy-second. As the penalty coefficient grows, which 120 rows are drawn affects the answer the model gives less. At penalty 1000 the thirty models say almost the same thing, because all of them stand close to the baseline. The bias column splits into two regions. The 34.039 at the unpenalized end is higher than expected: a system of equations with 29 columns on 120 rows is nearly singular and the solution swings so much that even the **mean** of the thirty models is corrupted. Up to penalty 100 this column falls to 30.976. Then it jumps to 35.295: there the penalty erases not only the noise but also the real region structure in the set. The point where the total column is smallest is penalty 100, and this is the same as the point where the validation column in the previous table was smallest. Two independent measurements — one the root error on a single split, the other the component decomposition on thirty samples — point to the same coefficient. #### From Which Set the Best Point Is Read Penalty 100 was read from the table, but **from which table** it was read is the question of the first lesson of the course. ```python PENALTY = [0.001, 0.01, 0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000] SCORES = [(c, rmse(fit(TRAIN, c), VAL), rmse(fit(TRAIN, c), TEST)) for c in PENALTY] by_val = min(SCORES, key=lambda t: t[1]) by_test = min(SCORES, key=lambda t: t[2]) print(f"best penalty if chosen on validation {by_val[0]:>3}: validation {by_val[1]:.3f}, " f"test {by_val[2]:.3f}") print(f"best penalty if chosen on test {by_test[0]:>3}: validation {by_test[1]:.3f}, " f"test {by_test[2]:.3f}") print(f"if chosen on test, reported number {by_test[2]:.3f}, " f"honest {by_val[2]:.3f}, difference {by_val[2] - by_test[2]:.3f} m3") print(f"\non the test set: baseline {rmse([0.0] * P, TEST):.3f}, unpenalized " f"{rmse(fit(TRAIN, 0.001), TEST):.3f}, chosen {by_val[2]:.3f}") print(f"gain from the penalty {rmse(fit(TRAIN, 0.001), TEST) - by_val[2]:.3f} m3, " f"gain of the model over the baseline {rmse([0.0] * P, TEST) - by_val[2]:.3f} m3") ``` ``` best penalty if chosen on validation 100: validation 5.553, test 5.559 best penalty if chosen on test 30: validation 5.556, test 5.534 if chosen on test, reported number 5.534, honest 5.559, difference 0.025 m3 on the test set: baseline 6.556, unpenalized 5.841, chosen 5.559 gain from the penalty 0.282 m3, gain of the model over the baseline 0.997 m3 ``` The two sets choose two different coefficients: validation 100, test 30. That both are defensible is the heart of the matter — the difference between them is not a real difference in quality but the own scatter of the two 252-row sets. But if the selection had been made on the test set, the reported number would have been 5.534; the honest number is 5.559. The difference is small at 0.025 m³ and **its sign is fixed**: the number on the set where the selection is made can never come out worse. Here there were eleven candidates; in the first lesson 48 candidates had raised the difference to 2 points. The share of selection grows with the number of candidates, but it is not zero even with zero candidates. The last two lines close the lesson's account. On the test set the baseline is 6.556 m³, the unpenalized linear model 5.841 m³, the regularized model 5.559 m³. The gain of the model over the baseline is 0.997 m³, that is, 15.2%. Of this, 0.282 m³, that is, 28.3%, comes from the penalty coefficient alone. Regularization did not add a new feature or teach a new rule; it chose how much of what had been learned would be kept, and took more than a quarter of the gain from there. #### Summary - When the penalty coefficient is raised from 0.001 to 1000, the weight magnitude falls from 10.040 to 0.601; the training error worsens without interruption from 4.603 to 5.453 m³, and this is the proof that the penalty is working. - The held-out error falls from 5.924 to 5.553 and returns to 5.932; the lowest point is at penalty 100 and, had the penalty been taken to infinity, the model would have turned into the baseline itself. - The decomposition shows the target of the penalty: the variance falls from 12.988 to 0.181, while the bias hits a floor at 30.976 and jumps to 35.295 at penalty 1000; there the penalty erases the real region structure. - The best penalty read from the validation set is 100, and it would have been 30 if read from the test set; choosing from the test would have lowered the reported number from 5.559 to 5.534, and the sign of this difference never turns around. - On the test set the baseline is 6.556, the unpenalized model 5.841, the regularized model 5.559 m³; the gain of the model over the baseline is 0.997 m³ and 28.3% of it comes from the penalty coefficient alone. #### Next Step In four lessons the separation has been established and the baseline has been written next to every number. In the setup of this lesson there is a decision that was silently made correctly: the scale of the columns was computed **from the training set only**. The decision can be reversed, and when it is reversed the held-out number improves — and the model has learned nothing. The next lesson applies four such corruptions one by one and prints how much each of them inflates the held-out number. The close of the course is there too. ### Data Leakage Source: https://academia.sh/en/courses/introduction-to-machine-learning/data-leakage Measuring how sources other than learning inflate the held-out number: against the honest line, which gives an accuracy of 0.7778 on the test set with a baseline of 0.7579, scaling before the split inflates this number by 0.0119, a feature that sees the future by 0.0754, a duplicated record by 0.0542 and a feature derived from the target by 0.1429; all four have the same sign. A model whose features have been replaced with six random columns reads an accuracy of 0.8757 on the held-out set while the duplicates are present and beats the baseline of 0.7857 by 9 points; on the rows whose twin it has not seen, it stays 9.2 points below the baseline. At the course wrap-up, the baseline, training and held-out numbers of eleven lessons are gathered in a single table. The previous lesson silently made one decision correctly: the scale of the columns was computed from the training set only. The decision can be reversed. When the scale is computed from the whole set, the rows of the held-out set contribute to producing the numbers the model will see, and the held-out set stops being held out by that amount. This is the quietest form of **data leakage**. Leakage is the mixing of information that will not be available at prediction time into the training or measurement path, and it always pushes the same way: the held-out number rises. This lesson applies the four kinds one by one, measures the inflation of each against its own honest counterpart and counts the second claim of the curriculum — **the difference does not always come from learning.** - **GE31.** The set, the label, the split, the seed and the measure are the same throughout the course. The held-out set is the test set: 252 rows. The honest line is the model of the first lesson: the depth 6 decision tree, baseline 0.7579, 0.8466 on training, 0.7778 on test. - **GE32.** The first leak is scaling by region: the region mean and scatter are computed from the training set alone in the honest form and from the whole set in the leaky form. - **GE33.** The second leak is the fourth-period reading and is **fictional**: it is formed after the label is written, for suspicious subscribers it comes out zero with probability 0.55, and if it does not, its value is 1.45 times as much. - **GE34.** The third leak is a join error: every subscriber record is produced three times, 1,260 subscribers rise to 3,780 rows and the split is made at the **row** level. - **GE35.** The fourth leak is the audit team assignment and is **fictional**: it is one with probability 0.88 for suspicious subscribers and 0.06 for the others, and it is recorded after the label. - **GE36.** The inflation is always measured against **its own honest counterpart**; the counterpart of the first leak is the honestly computed form of the same feature. - **GE37.** In the last section the features are replaced with six random columns. These columns are unrelated to the label and identical across the three copies of a subscriber. #### Four Leaks, Four Inflations The setup is the same as in the first lesson of the course and produces the same set with the same seed. ```python # generalization.py — MODEL. The regional measurement network (a fictional setup) of the Data Analytics curriculum is generated with the same # seed; readings are summarized at the subscriber level and a FICTIONAL illicit-use/fault suspicion label # is added on top. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) DATA = [] for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} for b in REGION: x["reg_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "reg_north", "reg_south", "reg_east", "reg_west", "reg_central"] def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, TEST = K[:756], K[1008:] def gini(s): p = sum(x["suspicious"] for x in s) / len(s) if s else 0.0 return 2 * p * (1 - p) def tree(s, depth, min_size=2): p = sum(x["suspicious"] for x in s) / len(s) best = None if depth and len(s) >= 2 * min_size and 0.0 < p < 1.0: for f in FIELDS: values = sorted({x[f] for x in s}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in s if x[f] < v] right = [x for x in s if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = gini(s) - (len(left) * gini(left) + len(right) * gini(right)) / len(s) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, min_size), "right": tree(best[4], depth - 1, min_size)} def predict(d, x): while "prediction" not in d: d = d["left"] if x[d["field"]] < d["threshold"] else d["right"] return d["prediction"] def accuracy(d, s): return sum(predict(d, x) == x["suspicious"] for x in s) / len(s) def baseline(train, s): c = int(sum(x["suspicious"] for x in train) * 2 > len(train)) return sum(x["suspicious"] == c for x in s) / len(s) ``` The four leaks are set up in turn and each is measured against its own honest counterpart. ```python BASE, RESULTS = list(FIELDS), [] for x in DATA: x["region_name"] = [b[0] for b in REGION if x["reg_" + b[0]]][0] def measure(fields, train, held, depth=6): global FIELDS FIELDS = fields m = tree(train, depth) FIELDS = BASE return baseline(train, held), accuracy(m, train), accuracy(m, held) def region_z(source): for b in {x["region_name"] for x in DATA}: v = [x["mean_consumption"] for x in source if x["region_name"] == b] o = sum(v) / len(v) s = max(1e-6, (sum((q - o) ** 2 for q in v) / len(v)) ** 0.5) for x in DATA: if x["region_name"] == b: x["region_z"] = round((x["mean_consumption"] - o) / s, 3) HONEST = measure(BASE, TRAIN, TEST) region_z(TRAIN) S1H = measure(BASE + ["region_z"], TRAIN, TEST) region_z(DATA) RESULTS.append(("1 scaling before the split", S1H, measure(BASE + ["region_z"], TRAIN, TEST))) for x in DATA: r, t = rng(SEED + 71000 + x["id"]), 0 for b in REGION: t = b[2] if x["region_name"] == b[0] else t x["fourth"] = 0.0 if (x["suspicious"] and r() < 0.55) else round( t * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 + 0.45 * x["suspicious"]), 2) RESULTS.append(("2 feature that sees the future", HONEST, measure(BASE + ["fourth"], TRAIN, TEST))) TWINS = [] for x in DATA: for c in range(3): y = dict(x) y["identity"], y["id"] = x["id"], x["id"] + 50000 * c TWINS.append(y) D = shuffle(TWINS, SEED + 93000) D_TRAIN, D_HELD = D[:int(len(D) * 0.6)], D[int(len(D) * 0.8):] TWIN_IDS = {x["identity"] for x in D_TRAIN} WITH_TWIN = [x for x in D_HELD if x["identity"] in TWIN_IDS] NO_TWIN = [x for x in D_HELD if x["identity"] not in TWIN_IDS] RESULTS.append(("3 duplicated record", HONEST, measure(BASE, D_TRAIN, D_HELD))) for x in DATA: r = rng(SEED + 81000 + x["id"]) x["audit"] = int(r() < (0.88 if x["suspicious"] else 0.06)) RESULTS.append(("4 feature derived from the target", HONEST, measure(BASE + ["audit"], TRAIN, TEST))) print(f"{'':<34} {'baseline':>8} {'train':>9} {'honest':>8} {'leaky':>10} {'inflation':>9}") print(f"{'honest line':<34} {HONEST[0]:>8.4f} {HONEST[1]:>9.4f} {HONEST[2]:>8.4f}") for name, h, s in RESULTS: print(f"{name:<34} {s[0]:>8.4f} {s[1]:>9.4f} {h[2]:>8.4f} {s[2]:>10.4f} " f"{s[2] - h[2]:>+9.4f}") print(f"\nduplicated set {len(D)} rows; held-out {len(D_HELD)}, with a twin in train " f"{len(WITH_TWIN)}, without {len(NO_TWIN)}") print(f"inside leak 3: on rows with a twin {measure(BASE, D_TRAIN, WITH_TWIN)[2]:.4f}, " f"on rows without a twin {measure(BASE, D_TRAIN, NO_TWIN)[2]:.4f}") ``` ``` baseline train honest leaky inflation honest line 0.7579 0.8466 0.7778 1 scaling before the split 0.7579 0.8492 0.7659 0.7778 +0.0119 2 feature that sees the future 0.7579 0.9312 0.7778 0.8532 +0.0754 3 duplicated record 0.7857 0.8646 0.7778 0.8320 +0.0542 4 feature derived from the target 0.7579 0.9590 0.7778 0.9206 +0.1429 duplicated set 3780 rows; held-out 756, with a twin in train 626, without 130 inside leak 3: on rows with a twin 0.8514, on rows without a twin 0.7385 ``` All four values of the inflation column **have the same sign**: +0.0119, +0.0754, +0.0542, +0.1429. This is not a coincidence but the definition of leakage: information that mixes into the held-out set always works in the model's favor. The constancy of the sign is the first mark that tells leakage from noise. The second column points to a trap. The training number also rises from 0.8466 to 0.9590, but this says nothing on its own — the second lesson showed that raising capacity raises the same column. The signature of leakage is the rise of **the held-out number**. The ranking is the reverse of the ranking of danger. The largest inflation is produced by the feature derived from the target: with 0.1429 it carries the model to 0.9206. But the name of that column is `audit` and it is likely that someone will ask "when is this column written". The smallest inflation comes from scaling before the split: 0.0119, that is, 3 subscribers. No column has been added, the code looks right and a difference of 1.2 points is not questioned in any report. The danger of leakage is inversely proportional not to its size but to its **visibility**. The third leak can be read inside itself. In the duplicated set, 626 of the 756 rows of the held-out set have their twin in the training set; the model gives 0.8514 on these rows and 0.7385 on the 130 rows that have no twin. The split separated the rows but did not separate the **subscriber**, and to see this by looking at the split ratio is impossible. In the second leak the fourth-period reading is a real measurement; the problem is not the wrongness of the measurement but its **time**. For each column in the set a single question is enough: is this value available at the moment the prediction will be given. #### A Model That Has Learned Nothing All four leaks inflated a real model. The next question is harsher: can leakage carry a model that has **learned nothing** above the baseline. The code below discards all the features of the duplicated set and puts six random columns in their place; nothing is left to learn. ```python for x in TWINS: r = rng(SEED + 31000 + x["identity"]) for j in range(6): x["g" + str(j)] = round(r(), 4) FIELDS = ["g" + str(j) for j in range(6)] print(f"{'capacity':>11} {'train':>9} {'held-out':>9} {'with twin':>9} {'no twin':>8}") print(f"{'baseline':>11} {baseline(D_TRAIN, D_TRAIN):>9.4f} {baseline(D_TRAIN, D_HELD):>9.4f} " f"{baseline(D_TRAIN, WITH_TWIN):>9.4f} {baseline(D_TRAIN, NO_TWIN):>8.4f}") for dp in (8, 12, 16): m = tree(D_TRAIN, dp) print(f"{'depth ' + str(dp):>11} {accuracy(m, D_TRAIN):>9.4f} {accuracy(m, D_HELD):>9.4f} " f"{accuracy(m, WITH_TWIN):>9.4f} {accuracy(m, NO_TWIN):>8.4f}") FIELDS = BASE ``` ``` capacity train held-out with twin no twin baseline 0.7787 0.7857 0.7796 0.8154 depth 8 0.8589 0.8228 0.8419 0.7308 depth 12 0.9528 0.8757 0.9073 0.7231 depth 16 0.9832 0.9061 0.9377 0.7538 ``` The depth 12 row is the measured form of the second claim of the curriculum. The model reads 0.8757 on the held-out set and beats the baseline of 0.7857 by **9.0 points**. If this number stands alone in a report it looks defensible: it has beaten the baseline, it has been measured on the held-out set, both conditions the course asks for are met. Yet the six columns the model looks at are random numbers. The difference shows in the last two columns. On the 626 rows whose twin is in training the model gives 0.9073, on the 130 rows that have no twin 0.7231, and the baseline of those 130 rows is 0.8154. On the rows where it has seen the twin the model recalls its memorization, and on the ones it has not seen it stays 9.2 points below the baseline. The only thing it has learned is which label sticks to certain combinations of six random numbers. The capacity column confirms this: the held-out number is 0.8228 at depth 8, 0.8757 at 12 and 0.9061 at 16. The curve of the second lesson works here **with the opposite sign**. This is the most useful auditing rule the course can give — that overfitting is not penalized on the held-out set is the most reliable sign that the set was not held out. The rule of the course is completed here: a performance measured on a set believed to be held out may also be memorization. The split itself is a decision to be audited, and the audit is made by looking not at the split ratio but at the split **unit** and at the **time** of each column. #### Summary - The inflation of all four kinds of leakage has the same sign: scaling before the split +0.0119, a feature that sees the future +0.0754, a duplicated record +0.0542, a feature derived from the target +0.1429. - The signature of leakage is the rise of the held-out number, not of the training number; capacity also raises the training number. - The size of the inflation is inversely proportional to its visibility; and because in a duplicated record the split separates the rows but not the subscriber, the accuracy is 0.8514 on the 626 rows whose twin is in training and 0.7385 on the 130 rows whose twin is not. - A model whose features have been replaced with six random columns reads 0.8757 on the held-out set and beats the baseline of 0.7857 by 9.0 points; on the rows whose twin it has not seen it stays 9.2 points below its baseline. - As capacity rises the leaky measurement improves (0.8228, 0.8757, 0.9061); that overfitting is not penalized on the held-out set is the most reliable sign that the set was not held out. #### Course Wrap-Up The eleven lessons stand in a single table; every number was read from its own lesson. The first six lessons use the 883/377 split, the last five the 756/252/252 split; the baseline therefore changes from row to row. | Lesson | Baseline | Training | Held-out set | |---|---|---|---| | Concept 01 What Is Machine Learning | most frequent class 0.7533 | tree 0.8709 | tree 0.8594 | | Concept 02 Types of Learning | most frequent class 0.7533 | supervised 0.8709 | supervised 0.8594 | | Concept 03 Self-Supervised Learning | most frequent class 0.7533 | with representation 0.8313 | with representation 0.8037 | | Concept 04 Application Areas | most frequent class 0.7533 | tree 0.8709 | tree 0.8594 | | Concept 05 Machine Learning Workflow | random order 14 illicit cases | ranking 30 illicit cases | ranking 30 illicit cases | | Concept 06 Roles | field procedure 13 illicit cases | — | model order 30 illicit cases | | Generalization 01 Training, Validation and Test Sets | most frequent class 0.7579 | depth 6 0.8466 | depth 6 0.7778 | | Generalization 02 Overfitting and Underfitting | most frequent class 0.8135 | depth 14 0.9352 | depth 14 0.7302 | | Generalization 03 Bias–Variance Tradeoff | mean 6.419 m³ | depth 3 5.105 m³ | depth 3 5.380 m³ | | Generalization 04 Regularization Intuition | mean 6.556 m³ | penalty 100 4.920 m³ | penalty 100 5.559 m³ | | Generalization 05 Data Leakage | most frequent class 0.7579 | leaky 0.9590 | leaky 0.9206 | The row of Concept 04 is the only classification one among the four problems in that lesson. In the row of Concept 06 the training column is empty: that lesson counts not the separation of training from the held-out set but the reading of the same held-out measure against three different baselines in three job definitions — 13, 23 and 21 illicit cases, gain 17, 7 and 9. The row of Generalization 05 is deliberately leaky. The difference between the third and the fourth column is **the rule of the course**: a performance not measured on a held-out set is not performance but memorization. In no row is the training number worse than the held-out one. The second column is **the rule of the curriculum**: a model is as good as the amount by which it beats a procedure dumber than itself, and a performance whose baseline is not written counts as unmeasured. The highest accuracy, 0.9206, says nothing; the lowest, 0.7302, says something, because it stays 8.3 points below its baseline. Eleven lessons established the separation and wrote the baseline next to every number. But the **features** given to the model were always taken as ready: the region had been turned into an indicator column and three readings into a mean from the start. Which column is to be turned into what kind of number, how much that conversion moves the difference from the baseline, and which conversion carries a silent leak like the one in this lesson were never asked. The next course, Data Preparation and Feature Engineering, takes up these questions.