--- title: 'Regression Metrics' source: 'https://academia.sh/en/courses/supervised-learning/regression-metrics' course: 'Supervised Learning' language: en updated: '2026-10-21T05:03:30+00:00' license: 'CC BY-SA 4.0' objectives: - 'Computes the mean absolute error, the mean squared error and the coefficient of determination against a baseline made of the train mean' - 'Shows that two members of the same family trade places on two measures and counts which rows the trade comes from' - 'Shows that the coefficient of determination is the form of the mean squared error read relative to the baseline: it does not produce a third ranking' --- # Regression Metrics The shift of the measure from counting to distance when the target is a number, and two members of the same family trading places on three measures: the linear model fitted to squared error leads on mean squared error with 22.3164, the one fitted to absolute error leads on mean absolute error with 3.6178. The baseline is the train mean (19.7665 m³) and the coefficient of determination is defined exactly relative to this baseline. 13 rows of the test set carry 37.59% of the squared error and 17.71% of the absolute error; when those 13 rows are trimmed, the ranking reverses on both measures. The number of candidates tried is three and no hyperparameter was swept. The previous four lessons built the measure for classification. The confusion matrix, precision, recall, the threshold and calibration all came out of a single operation: **counting**. How many examples fell into the right class, how many went to a false positive, how many were realized in a probability bin. There was something to count, because the prediction and the truth either coincided or did not. When the target is a number, there is no such thing as coinciding. For a subscriber who consumes 24.80 m³, saying 24.79 versus saying 11.20 is not a difference in count but a difference in **distance**, and how that distance is to be summed is a decision. This lesson measures that decision: two members of the same family, on the same held-out set, trade places on two measures. - **ET36.** The target is the subscriber's **mean consumption** (m³), computed from the readings spread over three periods. The set is the fictional subscriber table of the Introduction to Machine Learning course, the seed is 20260218, the split is 756/252/252. - **ET37.** The baseline is the **mean of the train set**: the same number is stated for every row. This is the regression counterpart of the "state the most frequent class" baseline in classification, and it does not learn. - **ET38.** The features stay fixed throughout the course: five region indicators, household size, satisfaction, the number of readings and the zero-reading indicator. The only thing that changes is the model. - **ET39.** The mean absolute error (MAE) is the mean of the absolute values of the errors, the mean squared error (MSE) is the mean of their squares. The MSE is in the square of the unit of the target; when its square root is taken it returns to the unit. - **ET40.** The coefficient of determination (R²) is computed as $1 - \text{MSE}_{\text{model}} / \text{MSE}_{\text{baseline}}$ and the denominator is **the baseline of ET37**. If the baseline changes, R² changes. - **ET41.** Both linear models use the same features and the same form; the only place they part is **which error they minimize during training**. Both are models. - **ET42.** The number of candidates tried in this lesson is **three** and no hyperparameter was swept; the difference inflated by the tuning budget is measured in `07`. ## Three Measures and the Baseline That Does Not Learn ```python # COMMON — MODEL: The FICTIONAL subscriber table of the Introduction to Machine Learning course # is rebuilt with the same seed; in this lesson the target is a number, not a class: # mean_consumption (m3). import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol = sum(v) / len(v), max(v) - min(v) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["area_north", "area_south", "area_east", "area_west", "area_central", "household", "satisfaction", "period", "zero_readings"] def mae(t, s): return sum(abs(t[i] - x["mean_consumption"]) for i, x in enumerate(s)) / len(s) def mse(t, s): return sum((t[i] - x["mean_consumption"]) ** 2 for i, x in enumerate(s)) / len(s) BASELINE = sum(x["mean_consumption"] for x in TRAIN) / len(TRAIN) BASELINE_MSE = mse([BASELINE] * len(TEST), TEST) def determination(t, s): return 1 - mse(t, s) / BASELINE_MSE print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}") print(f"train mean {BASELINE:.4f} m3, test mean " f"{sum(x['mean_consumption'] for x in TEST) / len(TEST):.4f} m3") print(f"baseline: MAE {mae([BASELINE] * len(TEST), TEST):.4f} " f"MSE {BASELINE_MSE:.4f} R2 {determination([BASELINE] * len(TEST), TEST):.4f}") ``` ``` train 756, validation 252, test 252 train mean 19.7665 m3, test mean 19.9234 m3 baseline: MAE 5.0806 MSE 42.6718 R2 0.0000 ``` The baseline states 19.7665 m³ and is off by 5.0806 m³ on average. The same error summed as squares is 42.6718; its square root is 6.53 m³. The gap between the two numbers comes from the measure itself: the sum of squares counts large errors more heavily than small ones, so when the error distribution is skewed the squared measure gives a larger number than the absolute one. The R² of the baseline is exactly zero by definition — the baseline explains no variation relative to itself. That the baseline is the **train** mean is not a detail but a necessity. The test set's own mean is not a prediction but a number read from the rows to be measured; making it the baseline would mean counting a procedure that has looked at the test set as the baseline. A procedure that does not learn must also see only the train set. ## Two Members of the Same Family The model is a linear model: each feature is multiplied by a coefficient and the products are summed. The only difference between the two members is which error they minimize during training. The first minimizes the squared error and solves the coefficients by least squares. The second minimizes the absolute error; it does this by lowering, in every round, the weight of the rows with large errors and rebuilding the same solution. ```python def solve(A, b): n = len(A) M = [list(A[i]) + [b[i]] for i in range(n)] for i in range(n): p = max(range(i, n), key=lambda r: abs(M[r][i])) M[i], M[p] = M[p], M[i] for r in range(n): if r != i: f = M[r][i] / M[i][i] for c in range(i, n + 1): M[r][c] -= f * M[i][c] return [M[i][n] / M[i][i] for i in range(n)] def weighted_least_squares(s, weights, penalty=1e-6): d = len(FIELDS) + 1 A = [[0.0] * d for _ in range(d)] b = [0.0] * d for x, g in zip(s, weights): v = [1.0] + [float(x[a]) for a in FIELDS] for i in range(d): b[i] += g * v[i] * x["mean_consumption"] for j in range(d): A[i][j] += g * v[i] * v[j] for i in range(1, d): A[i][i] += penalty return solve(A, b) def predict(w, x): return w[0] + sum(w[i + 1] * x[a] for i, a in enumerate(FIELDS)) def fit_absolute(s, rounds=40, floor_weight=0.05): w = weighted_least_squares(s, [1.0] * len(s)) for _ in range(rounds): g = [1.0 / max(abs(predict(w, x) - x["mean_consumption"]), floor_weight) for x in s] w = weighted_least_squares(s, g) return w REGION_MEAN = {} for x in TRAIN: REGION_MEAN.setdefault(x["region_name"], []).append(x["mean_consumption"]) REGION_MEAN = {k: sum(v) / len(v) for k, v in REGION_MEAN.items()} W_SQUARED, W_ABSOLUTE = weighted_least_squares(TRAIN, [1.0] * len(TRAIN)), fit_absolute(TRAIN) CANDIDATES = [("baseline", [BASELINE] * len(TEST)), ("regional mean", [REGION_MEAN[x["region_name"]] for x in TEST]), ("fitted to squared error", [predict(W_SQUARED, x) for x in TEST]), ("fitted to absolute error", [predict(W_ABSOLUTE, x) for x in TEST])] print(f"{'model':<26} {'MAE':>8} {'MSE':>9} {'R2':>8}") for name, t in CANDIDATES: print(f"{name:<26} {mae(t, TEST):>8.4f} {mse(t, TEST):>9.4f} {determination(t, TEST):>8.4f}") for measure, fn, direction in (("MAE", mae, 1), ("MSE", mse, 1), ("R2", determination, -1)): s = sorted(CANDIDATES[1:], key=lambda p: direction * fn(p[1], TEST)) print(f"{measure} ranking: " + ", ".join(f"{i + 1}) {a}" for i, (a, _) in enumerate(s))) ``` ``` model MAE MSE R2 baseline 5.0806 42.6718 0.0000 regional mean 4.0374 28.7760 0.3256 fitted to squared error 3.6504 22.3164 0.4770 fitted to absolute error 3.6178 22.3987 0.4751 MAE ranking: 1) fitted to absolute error, 2) fitted to squared error, 3) regional mean MSE ranking: 1) fitted to squared error, 2) fitted to absolute error, 3) regional mean R2 ranking: 1) fitted to squared error, 2) fitted to absolute error, 3) regional mean ``` The last three lines are the answer of the lesson. The mean absolute error reads the model fitted to absolute error as first: 3.6178 against 3.6504, a gap of 0.0326 m³. The mean squared error reads **another model** as first: 22.3164 against 22.3987, a gap of 0.0823. **Three candidates, two measures, two separate firsts.** No calculation is wrong; the two measures sum the same error list in two separate ways and produce two separate rankings. The regional mean is third on both measures and passes the baseline by a clear margin on both. Most of the difference the family adds is there: from 42.6718 the baseline drops to 28.7760 with the regional mean, and the linear model brings it down to 22.3164. The region alone provides 0.3256 of the explained variation, the remaining eight features provide 0.1514. The MSE difference of the two linear models is 0.0823. When the square roots are taken, this means 4.7240 m³ against 4.7327 m³; the typical error of the two models parts by 0.0087 m³. Because the raw value of the measure is in the square of the unit of the target, 0.0823 looks large to the eye, yet when returned to the unit it is of the same size class as the 0.0326 difference in the mean absolute error. Taking the square root of a squared measure before interpreting it is the cheapest way to see the true scale of the difference. ## Which Measure Weights Which Error The source of the trade is the weight the two measures give to large errors. The way to measure it is to sort the errors from largest to smallest and count which share of the total comes from which rows. ```python ERRORS = {name: sorted((abs(t[i] - x["mean_consumption"]) for i, x in enumerate(TEST)), reverse=True) for name, t in CANDIDATES[2:]} print(f"{'model':<26} {'top13 MAE':>10} {'top13 MSE':>10} " f"{'half MAE':>9} {'half MSE':>9}") for name, h in ERRORS.items(): def half(fn): total, running, count = sum(fn(q) for q in h), 0.0, 0 for q in h: running, count = running + fn(q), count + 1 if 2 * running >= total: return count print(f"{name:<26} {sum(h[:13]) / sum(h):>10.4f} " f"{sum(q * q for q in h[:13]) / sum(q * q for q in h):>10.4f} " f"{half(lambda q: q):>9} {half(lambda q: q * q):>9}") largest = sorted(range(len(TEST)), key=lambda i: -abs(CANDIDATES[2][1][i] - TEST[i]["mean_consumption"]))[:13] rest = [i for i in range(len(TEST)) if i not in set(largest)] subset = [TEST[i] for i in rest] print(f"\nafter trimming 13 rows ({len(subset)} rows)") for name, t in CANDIDATES[2:]: rest_pred = [t[i] for i in rest] print(f"{name:<26} MAE {mae(rest_pred, subset):.4f} MSE {mse(rest_pred, subset):.4f}") print(f"true values of the 13 trimmed rows: " f"{[TEST[i]['mean_consumption'] for i in largest[:4]]}") ``` ``` model top13 MAE top13 MSE half MAE half MSE fitted to squared error 0.1771 0.3759 58 22 fitted to absolute error 0.1838 0.3962 57 21 after trimming 13 rows (239 rows) fitted to squared error MAE 3.1674 MSE 14.6846 fitted to absolute error MAE 3.1186 MSE 14.3617 true values of the 13 trimmed rows: [43.48, 39.49, 36.53, 32.18] ``` **13** of the 252 rows of the test set, that is, five percent, carry 37.59% of the squared error; the same 13 rows carry only 17.71% of the absolute error. The same distinction shows up at the half of the total as well: half of the absolute error is spread over 58 rows, while half of the squared error is gathered in **22 rows**. The squared measure is read effectively from twenty-odd rows in a set of 252 rows. When these 13 rows are trimmed, the ranking reverses on both measures: the model fitted to absolute error moves ahead on both the MAE (3.1186 against 3.1674) and the MSE (14.3617 against 14.6846). The whole MSE advantage of the model fitted to squared error is in those 13 trimmed rows. The true values of the trimmed rows are 43.48, 39.49, 36.53, 32.18 m³ — the highest consumptions of the set. The model that minimizes the squared error accepts being off a little more on the remaining 239 rows in order to chase these extremes; the model that minimizes the absolute error does the opposite. What follows from this is that choosing a measure is a decision about **cost**. If, in a set of suspected illicit cases, one subscriber being off by 18 m³ and ten subscribers being off by 1.8 m³ each do not produce the same field visit cost, a measure that sums the two situations into the same number makes you choose the wrong model. ## The Coefficient of Determination Does Not Give a Third Ranking The R2 column gives **exactly the same** ranking as the MSE column, and this is not a coincidence. The definition is $1 - \text{MSE}_{\text{model}} / \text{MSE}_{\text{baseline}}$; the denominator is the same constant for all models (42.6718). Dividing by a constant cannot change a ranking. R² is not a third measure but **the form of the squared error read relative to the baseline**: 1 − 22.3164 / 42.6718 = 0.4770. What it adds is not a ranking but a **unit**. The MSE 22.3164 by itself does not say whether it is large or small; R² reads the same number as 0.4770, "it erased nearly half of the baseline's error". For this reason, when R² is written, **relative to which baseline** it was computed must also be written. Here the baseline is the mean of the train set. Had the denominator been computed from the test set's own mean, it would have been 42.6471 and R² would have come out as 0.4767 instead of 0.4770; the difference is small, because the two means are close to each other (19.7665 against 19.9234). When the train and test means are far apart, this difference does not stay small and the two R² values become incomparable. The number of candidates tried in this lesson is **three**: the regional mean, the linear model fitted to squared error, the linear model fitted to absolute error. No hyperparameter was swept in any of them, so no number in the table is inflated by the search. How the same table changes under a budget is the subject of `07`. ## Summary - When the target is a number, the measure moves from counting to distance and how the distance is to be summed is a decision; the baseline is the train mean (19.7665 m³; MAE 5.0806, MSE 42.6718). - Two members of the same family trade places on two measures: the model fitted to absolute error is first on the MAE with 3.6178, the model fitted to squared error is first on the MSE with 22.3164. - 13 rows of the test set carry 37.59% of the squared error and 17.71% of the absolute error; half of the squared error comes from 22 rows, half of the absolute error from 58 rows. - When those 13 rows are trimmed, the ranking reverses on both measures (3.1186 against 3.1674 and 14.3617 against 14.6846); the whole advantage of the squared measure is in those rows. - The coefficient of determination produces no third ranking; because its denominator is constant for all models, it is the form of the mean squared error read relative to the baseline and cannot be written without a baseline. ## Next Step All the numbers of this lesson were read from a single split: 756 rows of train, 252 rows of test. The gap between the two linear models came out as 0.0326 on the MAE and 0.0823 on the MSE. These are small numbers and so far there is no measure of fluctuation next to them — whether another split would give the same ranking is not known. The next lesson treats the split itself as a tuning choice: into how many folds the remaining data is split, how much the estimate fluctuates, how many trainings are done as the number of folds grows, and why the extreme that leaves a single row in each fold is not always better.