--- title: 'Baseline Model' source: 'https://academia.sh/en/courses/model-ethics-and-interpretation/baseline-model' course: 'Model Evaluation, Interpretability and Ethics' language: en updated: '2026-10-21T05:03:26+00:00' license: 'CC BY-SA 4.0' objectives: - 'Builds eight separate baseline candidates on the same problem and computes the range over which the difference of the same model moves' - 'Shows that the baseline has a capacity of its own: as the number of lookup-table cells grows, the train and test numbers part ways' - 'Counts how many subgroups the choice of baseline pushes below the baseline and compares the baseline choices in the seven courses of the Machine Learning and Data Science curriculum' --- # Baseline Model Measuring that the baseline is not a tool but a design object: the same depth-6 decision tree reads between -0.0119 and +0.5357 against eight baseline candidates, and the choice alone moves the number by 0.5476. A lookup-table baseline has a capacity of its own — with five cells it ties the model at 0.7579, with fifteen cells it beats the model at 0.7897, with thirty-nine cells it drops to 0.7500. The choice of baseline changes direction in the subgroups too: the model that falls below the most-frequent-class baseline in four of twelve subgroups falls below the region-and-tier lookup table in eight, and the difference gap shrinks from 0.2173 to 0.1351. The baselines chosen in the seven courses of the Machine Learning and Data Science curriculum are collected in a single table. The previous lesson turned the baseline into a question of its own for the second time. When the cost matrix changed, the baseline changed too: in one, flagging nobody, in the other, flagging everybody. Across the subgroups every group turned out to have its own baseline, which was 0.0000 in one group and 0.7882 in another. Over seven courses the baseline was a **tool** — a fixed mark that makes the model's number readable and is not argued over again once it is chosen. This lesson measures that it is not fixed. How many baselines a problem has, how one is chosen among them, in which two separate ways the choice lets the same model be read. Here the baseline is not a tool but a **design object**: a procedure with a capacity of its own, an overfitting of its own and a subgroup behavior of its own. At the end of the lesson the baselines actually chosen in the seven courses of the Machine Learning and Data Science curriculum are collected in a single table. - **ED16.** The setup, the split (**756/252/252**), the seed, the metric (**accuracy**) and the subgroup definitions carry over unchanged from the previous two lessons: four axes, **13 subgroups**, **a group whose n is below 25 does not enter the gap**. - **ED17.** The model being compared is the **depth-6 decision tree**, and it gives **0.7778** on the test set. This number never changes during the lesson; the only thing that changes is the baseline placed opposite it. - **ED18.** A baseline candidate is a procedure that, even if learned from the training set, does not look at all of the features the model uses and does not itself run a tuning search. The boundary is a decision. - **ED19.** Eight candidates are tested: calling everyone suspicious, random assignment by class proportion, stating the most frequent class, a single-split rule, and four **lookup tables** — region; region and tier; those plus household; all four axes. A lookup table states the training majority in a cell, and the most frequent class in a cell it has not seen; **the number of cells is written**, because that is what is being measured. - **ED20.** Random assignment is **the mean of twenty runs**; if a single run were printed the number would swing from run to run. - **ED21.** No choice among the candidates is **made** in this lesson. What is measured is not the result of the choice but how far the choice can move the same model. - **ED22.** Every number in the course table of the last section was read from its source lesson; the units are not shared across courses and the rows are not added to one another. ## How Many Baselines a Problem Has The first block sets up the scenario, trains the model and measures the eight baseline candidates on the same test set. ```python # MODEL: the measurement network of the Data Analytics curriculum is a FICTIONAL setup generated with the # same seed; readings are summarized at subscriber level and a FICTIONAL illicit-use/fault suspicion label is added on top. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, READINGS, DATA = [], {}, [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() v = READINGS.get(k["id"]) if v is None: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] def three_way_split(rows): a, b = int(len(rows) * 0.6), int(len(rows) * 0.8) return rows[:a], rows[a:b], rows[b:] def impurity(rows): p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0 return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): p, best = sum(x["suspicious"] for x in rows) / len(rows), None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = impurity(rows) - (len(left) * impurity(left) + len(right) * impurity(right)) / len(rows) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields), "right": tree(best[4], depth - 1, fields)} def predict(node, x): while "prediction" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["prediction"] def accuracy(node, rows): return sum(predict(node, x) == x["suspicious"] for x in rows) / len(rows) def subgroups(x): # four axes, thirteen subgroups; carried over unchanged from lesson 01 return [("region", x["region_name"]), ("tariff", "10" if x["mean_consumption"] <= 10 else ("25" if x["mean_consumption"] <= 25 else "40")), ("household", "1-2" if x["household"] <= 2 else ("3-4" if x["household"] <= 4 else "5+")), ("reading", "few" if x["period"] <= 2 else "full")] SUBGROUPS = sorted({g for x in DATA for g in subgroups(x)}) TRAIN, VAL, TEST = three_way_split(shuffle(DATA, SEED + 90000)) M, STUMP = tree(TRAIN, 6, FIELDS), tree(TRAIN, 1, FIELDS) RATE = sum(x["suspicious"] for x in TRAIN) / len(TRAIN) MAJORITY = int(RATE > 0.5) def groups_of(x): return dict(subgroups(x)) def lookup(key_fn): # lookup-table baseline: the training majority in the cell counts = {} for x in TRAIN: counts.setdefault(key_fn(x), [0, 0])[x["suspicious"]] += 1 votes = {c: int(v[1] > v[0]) for c, v in counts.items()} return lambda x: votes.get(key_fn(x), MAJORITY) def random_assign(rows, n=20): # assignment by class proportion, the mean of twenty runs t = 0.0 for j in range(n): r = rng(SEED + 61000 + j) t += sum((1 if r() < RATE else 0) == x["suspicious"] for x in rows) / len(rows) return t / n def hit_rate(f, rows): return sum(f(x) == x["suspicious"] for x in rows) / len(rows) KEYS = (("region", lambda x: x["region_name"]), ("region+tier", lambda x: (x["region_name"], groups_of(x)["tariff"])), ("+household", lambda x: (x["region_name"], groups_of(x)["tariff"], groups_of(x)["household"])), ("four axes", lambda x: (x["region_name"], groups_of(x)["tariff"], groups_of(x)["household"], groups_of(x)["reading"]))) CANDIDATES = [("all suspicious", 1, lambda x: 1), ("most frequent class", 1, lambda x: MAJORITY), ("single rule", 2, lambda x: predict(STUMP, x))] CANDIDATES += [(name, len({k(x) for x in TRAIN}), lookup(k)) for name, k in KEYS] model_acc = accuracy(M, TEST) print(f"model, depth-6 decision tree: train {accuracy(M, TRAIN):.4f} test {model_acc:.4f}\n\n" f"{'baseline candidate':<19} {'cells':>5} {'train':>7} {'test':>7} {'model diff':>12}") print(f"{'random assignment':<19} {'--':>5} {random_assign(TRAIN):>7.4f} {random_assign(TEST):>7.4f} " f"{model_acc - random_assign(TEST):>+12.4f}") diffs = [model_acc - random_assign(TEST)] for name, n, f in CANDIDATES: diffs.append(model_acc - hit_rate(f, TEST)) print(f"{name:<19} {n:>5} {hit_rate(f, TRAIN):>7.4f} {hit_rate(f, TEST):>7.4f} {diffs[-1]:>+12.4f}") print(f"\nthe baseline choice makes the same model read between {min(diffs):+.4f} and {max(diffs):+.4f}:" f" range {max(diffs) - min(diffs):.4f}") ``` ``` model, depth-6 decision tree: train 0.8466 test 0.7778 baseline candidate cells train test model diff random assignment -- 0.6522 0.6450 +0.1327 all suspicious 1 0.2222 0.2421 +0.5357 most frequent class 1 0.7778 0.7579 +0.0198 single rule 2 0.7778 0.7579 +0.0198 region 5 0.7778 0.7579 +0.0198 region+tier 15 0.7897 0.7897 -0.0119 +household 39 0.7976 0.7500 +0.0278 four axes 66 0.8095 0.7738 +0.0040 the baseline choice makes the same model read between -0.0119 and +0.5357: range 0.5476 ``` The last column is eight separate readings of a single model. The model did not change at all: 0.7778 on the test set. But against the "call everyone suspicious" baseline the model's contribution is **+0.5357**, against the "region and tier lookup table" baseline it is **−0.0119**. The **0.5476** between them is a number produced only by **the choice of baseline**, not by the data or the model. The same piece of work can be told in two separate reports, and both are computed correctly. The three middle rows give the second result. The most frequent class, a single-split rule and a lookup table that looks at the region produce **the same number**: 0.7579. Because the suspicious share does not exceed half in any region, all five cells of the region table say the same answer. Making a baseline more complicated does not make it stronger; it gets stronger only if the majority **changes** in the cells. The main finding is two rows further down. When region and tariff tier are taken together, the majority flips in some of the fifteen cells and the table reads 0.7897 — **above the depth-6 decision tree**. A tree trained on eleven features, which tried hundreds of candidate splits, loses to a lookup table with two columns. When household is added the cell count rises to 39: the train number climbs to 0.7976 while the test number falls to 0.7500. With all four axes there are 66 cells, train 0.8095, test 0.7738. **A baseline has a capacity too, and when that capacity is exceeded the baseline overfits as well.** This is what lifts the baseline out of being a tool and turns it into a design object: when it is chosen, the questions asked when choosing a model are asked. ## The Choice of Baseline Changes Direction in the Subgroups The overall difference between the two baselines is only 0.0318. Once we go down to the subgroups, the two do not say the same thing. ```python REGION_TIER = lookup(lambda x: (x["region_name"], groups_of(x)["tariff"])) print(f"{'subgroup':<14} {'n':>4} {'model':>7} {'A baseline':>10} {'A diff':>8} " f"{'B baseline':>10} {'B diff':>8}") diffs_a, diffs_b = [], [] for g in SUBGROUPS: rows = [x for x in TEST if g in subgroups(x)] base_a, base_b, acc_g = hit_rate(lambda x: MAJORITY, rows), hit_rate(REGION_TIER, rows), accuracy(M, rows) if len(rows) >= 25: diffs_a.append(acc_g - base_a) diffs_b.append(acc_g - base_b) print(f"{' '.join(g):<14} {len(rows):>4} {acc_g:>7.4f} {base_a:>10.4f} {acc_g - base_a:>+8.4f} " f"{base_b:>10.4f} {acc_g - base_b:>+8.4f}{' n<25' if len(rows) < 25 else ''}") print(f"\nA most frequent class: diff {min(diffs_a):+.4f} .. {max(diffs_a):+.4f}, gap " f"{max(diffs_a) - min(diffs_a):.4f}, below baseline {sum(1 for v in diffs_a if v < 0)}/{len(diffs_a)}") print(f"B region+tier : diff {min(diffs_b):+.4f} .. {max(diffs_b):+.4f}, gap " f"{max(diffs_b) - min(diffs_b):.4f}, below baseline {sum(1 for v in diffs_b if v < 0)}/{len(diffs_b)}") ``` ``` subgroup n model A baseline A diff B baseline B diff household 1-2 73 0.7945 0.7534 +0.0411 0.8219 -0.0274 household 3-4 143 0.7762 0.7692 +0.0070 0.7832 -0.0070 household 5+ 36 0.7500 0.7222 +0.0278 0.7500 +0.0000 reading few 85 0.7412 0.7882 -0.0471 0.7647 -0.0235 reading full 167 0.7964 0.7425 +0.0539 0.8024 -0.0060 region central 58 0.6724 0.6552 +0.0172 0.6552 +0.0172 region east 36 0.6111 0.4722 +0.1389 0.6944 -0.0833 region north 68 0.7794 0.8235 -0.0441 0.8235 -0.0441 region south 58 0.8793 0.8276 +0.0517 0.8276 +0.0517 region west 32 0.9688 1.0000 -0.0312 1.0000 -0.0312 tariff 10 12 0.5833 0.6667 -0.0833 0.6667 -0.0833 n<25 tariff 25 193 0.8290 0.8394 -0.0104 0.8394 -0.0104 tariff 40 47 0.6170 0.4468 +0.1702 0.6170 +0.0000 A most frequent class: diff -0.0471 .. +0.1702, gap 0.2173, below baseline 4/12 B region+tier : diff -0.0833 .. +0.0517, gap 0.1351, below baseline 8/12 ``` The east region is where the two baselines part. Against the most frequent class the model beats the baseline there by **0.1389** and gives one of the best contributions of the course; against the region and tier table it stays **0.0833 below** the baseline. The same happens in the tariff 40 tier: 0.0000 instead of +0.1702. Both numbers are correct. The difference is that the first baseline is unaware of the east region's high suspicion rate, while the second already knows that rate. A model can say what it adds in a subgroup only when it is compared with a baseline that **already knows** that subgroup. The two summary lines give the result. Against the most-frequent-class baseline the model is below the baseline in **four** of the twelve subgroups; against the lookup-table baseline, in **eight**. The difference gap also falls from 0.2173 to 0.1351. A strong baseline both shrinks the model's contribution and flattens it across subgroups; this is not a flaw but the consequence of the baseline already taking the subgroup into account. Which baseline is reported is therefore the decision that determines what the study claims. ## Baselines Chosen in the Seven Courses of Machine Learning and Data Science Over seven courses every lesson chose a baseline and wrote down its reason. The table below collects those choices from the source lessons. The units are not shared and the rows are not added to one another. | Course | Chosen procedure that learns nothing | Unit | Example number | |---|---|---|---| | Introduction to Machine Learning | the most frequent class of the training set | accuracy | 0.7579 on the test set | | Data Preparation and Feature Engineering | the same setup with the preparation decision not applied | accuracy | between 0.7302 and 0.8294 | | Supervised Learning | the most frequent class; stating the mean in regression | accuracy, m³ error | 0.7579 and 6.5324 m³ on the test set | | Unsupervised Learning | random assignment, column mean, a single cluster | within-cluster distance, reconstruction error | random assignment 5.9790 | | Deep Learning | a separate most frequent class for each dataset | accuracy | subscriber 0.7579; grid 0.5850; sequence 0.7100 | | Natural Language Processing and Computer Vision | a domain-specific baseline | accuracy, overlap, vocabulary length | twelve baselines in ratio units between 0.0400 and 0.8731 | | Reinforcement Learning and Recommender Systems | random action; the most-chosen item | return; hit within the first five | 10.432 and 0.6527 | Two things can be read from the table. First, the baseline **changes type** from course to course: stating a class, stating a mean, behaving randomly, recommending the most chosen, or putting in the state where a decision was not applied. Second, the rows of the Deep Learning course and of the Natural Language Processing and Computer Vision course show that even within one course there is not a single baseline: four separate datasets, four separate baselines. Whenever the sentence "it beat the baseline" was written, what was beaten was a different procedure each time, and the table of this lesson gives the size of that difference. ## Summary - The baseline is not a tool but a design object. The same depth-6 decision tree reads between **−0.0119 and +0.5357** against eight baseline candidates; the choice alone moves the number by **0.5476**. - Making a baseline more complicated does not make it stronger: the most frequent class, the single-split rule and the region lookup table give the same 0.7579, because the majority flips in no region. - The baseline has a capacity too. The lookup table **beats the model** with 0.7897 in 15 cells, and in 39 cells it climbs to 0.7976 on train while falling to 0.7500 on test: the baseline overfits as well. - The choice of baseline changes direction in the subgroups. In the east region the model is +0.1389 against one baseline and −0.0833 against the other. The model that is below the baseline in 4 of 12 subgroups against the most frequent class is below it in **8** against the lookup table; the difference gap falls from 0.2173 to 0.1351. - In the seven courses of the Machine Learning and Data Science curriculum the baseline changed type, and even within a single course more than one baseline was used; in the sentence "it beat the baseline" what is beaten is a different procedure each time. ## Next Step Over three lessons fourteen separate number pairs were compared, and in all of them the difference was read directly: 0.0198, −0.0119, +0.1702. What was never asked in any of them is whether these differences can be told apart from **the noise of a 252-subscriber set**. A contribution of 0.1389 read from 36 subscribers in one subgroup could vanish if the class of a few subscribers flipped. Hypothesis testing, permutation testing and the bootstrap were measured in the Statistical Analysis and Exploratory Data Analysis course and will not be repeated. The next lesson measures only this: how large the **margin** of the difference between two models is, how much narrower a margin the **paired comparison** measured on the same samples gives than the independent comparison, and how the size of the held-out set determines that margin.