Regression Metrics

The previous four lessons built the measure for classification. The confusion matrix, precision, recall, the threshold and calibration all came out of a single operation: counting. How many examples fell into the right class, how many went to a false positive, how many were realized in a probability bin. There was something to count, because the prediction and the truth either coincided or did not.

When the target is a number, there is no such thing as coinciding. For a subscriber who consumes 24.80 m³, saying 24.79 versus saying 11.20 is not a difference in count but a difference in distance, and how that distance is to be summed is a decision. This lesson measures that decision: two members of the same family, on the same held-out set, trade places on two measures.

• ET36. The target is the subscriber’s mean consumption (m³), computed from the readings spread over three periods. The set is the fictional subscriber table of the Introduction to Machine Learning course, the seed is 20260218, the split is 756/252/252.
• ET37. The baseline is the mean of the train set: the same number is stated for every row. This is the regression counterpart of the “state the most frequent class” baseline in classification, and it does not learn.
• ET38. The features stay fixed throughout the course: five region indicators, household size, satisfaction, the number of readings and the zero-reading indicator. The only thing that changes is the model.
• ET39. The mean absolute error (MAE) is the mean of the absolute values of the errors, the mean squared error (MSE) is the mean of their squares. The MSE is in the square of the unit of the target; when its square root is taken it returns to the unit.
• ET40. The coefficient of determination (R²) is computed as $1 - \text{MSE}_{\text{model}} / \text{MSE}_{\text{baseline}}$ and the denominator is the baseline of ET37. If the baseline changes, R² changes.
• ET41. Both linear models use the same features and the same form; the only place they part is which error they minimize during training. Both are models.
• ET42. The number of candidates tried in this lesson is three and no hyperparameter was swept; the difference inflated by the tuning budget is measured in 07.

Three Measures and the Baseline That Does Not Learn

# COMMON — MODEL: The FICTIONAL subscriber table of the Introduction to Machine Learning course
# is rebuilt with the same seed; in this lesson the target is a number, not a class:
# mean_consumption (m3).
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, DATA = [], []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": rating})
for k in SUBSCRIBERS:
    r, v = rng(SEED + 7000 + k["id"]), []
    for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        v.append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - d * 0.05) * 100 + 0.5) / 100)
        r()
    if not v:
        continue
    mean, vol = sum(v) / len(v), max(v) - min(v)
    x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"],
         "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "region_name": k["region"][0]}
    for b in REGION:
        x["area_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)


def shuffle(data, seed):
    r, s = rng(seed), list(range(len(data)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [data[i] for i in s]


K = shuffle(DATA, SEED + 90000)
TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:]
FIELDS = ["area_north", "area_south", "area_east", "area_west", "area_central",
          "household", "satisfaction", "period", "zero_readings"]


def mae(t, s):
    return sum(abs(t[i] - x["mean_consumption"]) for i, x in enumerate(s)) / len(s)


def mse(t, s):
    return sum((t[i] - x["mean_consumption"]) ** 2 for i, x in enumerate(s)) / len(s)


BASELINE = sum(x["mean_consumption"] for x in TRAIN) / len(TRAIN)
BASELINE_MSE = mse([BASELINE] * len(TEST), TEST)


def determination(t, s):
    return 1 - mse(t, s) / BASELINE_MSE


print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}")
print(f"train mean {BASELINE:.4f} m3, test mean "
      f"{sum(x['mean_consumption'] for x in TEST) / len(TEST):.4f} m3")
print(f"baseline: MAE {mae([BASELINE] * len(TEST), TEST):.4f}  "
      f"MSE {BASELINE_MSE:.4f}  R2 {determination([BASELINE] * len(TEST), TEST):.4f}")

train 756, validation 252, test 252
train mean 19.7665 m3, test mean 19.9234 m3
baseline: MAE 5.0806  MSE 42.6718  R2 0.0000

The baseline states 19.7665 m³ and is off by 5.0806 m³ on average. The same error summed as squares is 42.6718; its square root is 6.53 m³. The gap between the two numbers comes from the measure itself: the sum of squares counts large errors more heavily than small ones, so when the error distribution is skewed the squared measure gives a larger number than the absolute one. The R² of the baseline is exactly zero by definition — the baseline explains no variation relative to itself.

That the baseline is the train mean is not a detail but a necessity. The test set’s own mean is not a prediction but a number read from the rows to be measured; making it the baseline would mean counting a procedure that has looked at the test set as the baseline. A procedure that does not learn must also see only the train set.

Two Members of the Same Family

The model is a linear model: each feature is multiplied by a coefficient and the products are summed. The only difference between the two members is which error they minimize during training. The first minimizes the squared error and solves the coefficients by least squares. The second minimizes the absolute error; it does this by lowering, in every round, the weight of the rows with large errors and rebuilding the same solution.

def solve(A, b):
    n = len(A)
    M = [list(A[i]) + [b[i]] for i in range(n)]
    for i in range(n):
        p = max(range(i, n), key=lambda r: abs(M[r][i]))
        M[i], M[p] = M[p], M[i]
        for r in range(n):
            if r != i:
                f = M[r][i] / M[i][i]
                for c in range(i, n + 1):
                    M[r][c] -= f * M[i][c]
    return [M[i][n] / M[i][i] for i in range(n)]


def weighted_least_squares(s, weights, penalty=1e-6):
    d = len(FIELDS) + 1
    A = [[0.0] * d for _ in range(d)]
    b = [0.0] * d
    for x, g in zip(s, weights):
        v = [1.0] + [float(x[a]) for a in FIELDS]
        for i in range(d):
            b[i] += g * v[i] * x["mean_consumption"]
            for j in range(d):
                A[i][j] += g * v[i] * v[j]
    for i in range(1, d):
        A[i][i] += penalty
    return solve(A, b)


def predict(w, x):
    return w[0] + sum(w[i + 1] * x[a] for i, a in enumerate(FIELDS))


def fit_absolute(s, rounds=40, floor_weight=0.05):
    w = weighted_least_squares(s, [1.0] * len(s))
    for _ in range(rounds):
        g = [1.0 / max(abs(predict(w, x) - x["mean_consumption"]), floor_weight) for x in s]
        w = weighted_least_squares(s, g)
    return w


REGION_MEAN = {}
for x in TRAIN:
    REGION_MEAN.setdefault(x["region_name"], []).append(x["mean_consumption"])
REGION_MEAN = {k: sum(v) / len(v) for k, v in REGION_MEAN.items()}

W_SQUARED, W_ABSOLUTE = weighted_least_squares(TRAIN, [1.0] * len(TRAIN)), fit_absolute(TRAIN)
CANDIDATES = [("baseline", [BASELINE] * len(TEST)),
              ("regional mean", [REGION_MEAN[x["region_name"]] for x in TEST]),
              ("fitted to squared error", [predict(W_SQUARED, x) for x in TEST]),
              ("fitted to absolute error", [predict(W_ABSOLUTE, x) for x in TEST])]

print(f"{'model':<26} {'MAE':>8} {'MSE':>9} {'R2':>8}")
for name, t in CANDIDATES:
    print(f"{name:<26} {mae(t, TEST):>8.4f} {mse(t, TEST):>9.4f} {determination(t, TEST):>8.4f}")
for measure, fn, direction in (("MAE", mae, 1), ("MSE", mse, 1), ("R2", determination, -1)):
    s = sorted(CANDIDATES[1:], key=lambda p: direction * fn(p[1], TEST))
    print(f"{measure} ranking: "
          + ", ".join(f"{i + 1}) {a}" for i, (a, _) in enumerate(s)))

model                           MAE       MSE       R2
baseline                     5.0806   42.6718   0.0000
regional mean                4.0374   28.7760   0.3256
fitted to squared error      3.6504   22.3164   0.4770
fitted to absolute error     3.6178   22.3987   0.4751
MAE ranking: 1) fitted to absolute error, 2) fitted to squared error, 3) regional mean
MSE ranking: 1) fitted to squared error, 2) fitted to absolute error, 3) regional mean
R2 ranking: 1) fitted to squared error, 2) fitted to absolute error, 3) regional mean

The last three lines are the answer of the lesson. The mean absolute error reads the model fitted to absolute error as first: 3.6178 against 3.6504, a gap of 0.0326 m³. The mean squared error reads another model as first: 22.3164 against 22.3987, a gap of 0.0823. Three candidates, two measures, two separate firsts. No calculation is wrong; the two measures sum the same error list in two separate ways and produce two separate rankings.

The regional mean is third on both measures and passes the baseline by a clear margin on both. Most of the difference the family adds is there: from 42.6718 the baseline drops to 28.7760 with the regional mean, and the linear model brings it down to 22.3164. The region alone provides 0.3256 of the explained variation, the remaining eight features provide 0.1514.

The MSE difference of the two linear models is 0.0823. When the square roots are taken, this means 4.7240 m³ against 4.7327 m³; the typical error of the two models parts by 0.0087 m³. Because the raw value of the measure is in the square of the unit of the target, 0.0823 looks large to the eye, yet when returned to the unit it is of the same size class as the 0.0326 difference in the mean absolute error. Taking the square root of a squared measure before interpreting it is the cheapest way to see the true scale of the difference.

Which Measure Weights Which Error

The source of the trade is the weight the two measures give to large errors. The way to measure it is to sort the errors from largest to smallest and count which share of the total comes from which rows.

ERRORS = {name: sorted((abs(t[i] - x["mean_consumption"]) for i, x in enumerate(TEST)),
                       reverse=True) for name, t in CANDIDATES[2:]}
print(f"{'model':<26} {'top13 MAE':>10} {'top13 MSE':>10} "
      f"{'half MAE':>9} {'half MSE':>9}")
for name, h in ERRORS.items():
    def half(fn):
        total, running, count = sum(fn(q) for q in h), 0.0, 0
        for q in h:
            running, count = running + fn(q), count + 1
            if 2 * running >= total:
                return count
    print(f"{name:<26} {sum(h[:13]) / sum(h):>10.4f} "
          f"{sum(q * q for q in h[:13]) / sum(q * q for q in h):>10.4f} "
          f"{half(lambda q: q):>9} {half(lambda q: q * q):>9}")

largest = sorted(range(len(TEST)),
                 key=lambda i: -abs(CANDIDATES[2][1][i] - TEST[i]["mean_consumption"]))[:13]
rest = [i for i in range(len(TEST)) if i not in set(largest)]
subset = [TEST[i] for i in rest]
print(f"\nafter trimming 13 rows ({len(subset)} rows)")
for name, t in CANDIDATES[2:]:
    rest_pred = [t[i] for i in rest]
    print(f"{name:<26} MAE {mae(rest_pred, subset):.4f}  MSE {mse(rest_pred, subset):.4f}")
print(f"true values of the 13 trimmed rows: "
      f"{[TEST[i]['mean_consumption'] for i in largest[:4]]}")

model                       top13 MAE  top13 MSE  half MAE  half MSE
fitted to squared error        0.1771     0.3759        58        22
fitted to absolute error       0.1838     0.3962        57        21

after trimming 13 rows (239 rows)
fitted to squared error    MAE 3.1674  MSE 14.6846
fitted to absolute error   MAE 3.1186  MSE 14.3617
true values of the 13 trimmed rows: [43.48, 39.49, 36.53, 32.18]

13 of the 252 rows of the test set, that is, five percent, carry 37.59% of the squared error; the same 13 rows carry only 17.71% of the absolute error. The same distinction shows up at the half of the total as well: half of the absolute error is spread over 58 rows, while half of the squared error is gathered in 22 rows. The squared measure is read effectively from twenty-odd rows in a set of 252 rows.

When these 13 rows are trimmed, the ranking reverses on both measures: the model fitted to absolute error moves ahead on both the MAE (3.1186 against 3.1674) and the MSE (14.3617 against 14.6846). The whole MSE advantage of the model fitted to squared error is in those 13 trimmed rows. The true values of the trimmed rows are 43.48, 39.49, 36.53, 32.18 m³ — the highest consumptions of the set. The model that minimizes the squared error accepts being off a little more on the remaining 239 rows in order to chase these extremes; the model that minimizes the absolute error does the opposite.

What follows from this is that choosing a measure is a decision about cost. If, in a set of suspected illicit cases, one subscriber being off by 18 m³ and ten subscribers being off by 1.8 m³ each do not produce the same field visit cost, a measure that sums the two situations into the same number makes you choose the wrong model.

The Coefficient of Determination Does Not Give a Third Ranking

The R2 column gives exactly the same ranking as the MSE column, and this is not a coincidence. The definition is $1 - \text{MSE}_{\text{model}} / \text{MSE}_{\text{baseline}}$; the denominator is the same constant for all models (42.6718). Dividing by a constant cannot change a ranking. R² is not a third measure but the form of the squared error read relative to the baseline: 1 − 22.3164 / 42.6718 = 0.4770.

What it adds is not a ranking but a unit. The MSE 22.3164 by itself does not say whether it is large or small; R² reads the same number as 0.4770, “it erased nearly half of the baseline’s error”. For this reason, when R² is written, relative to which baseline it was computed must also be written. Here the baseline is the mean of the train set. Had the denominator been computed from the test set’s own mean, it would have been 42.6471 and R² would have come out as 0.4767 instead of 0.4770; the difference is small, because the two means are close to each other (19.7665 against 19.9234). When the train and test means are far apart, this difference does not stay small and the two R² values become incomparable.

The number of candidates tried in this lesson is three: the regional mean, the linear model fitted to squared error, the linear model fitted to absolute error. No hyperparameter was swept in any of them, so no number in the table is inflated by the search. How the same table changes under a budget is the subject of 07.

Summary

• When the target is a number, the measure moves from counting to distance and how the distance is to be summed is a decision; the baseline is the train mean (19.7665 m³; MAE 5.0806, MSE 42.6718).
• Two members of the same family trade places on two measures: the model fitted to absolute error is first on the MAE with 3.6178, the model fitted to squared error is first on the MSE with 22.3164.
• 13 rows of the test set carry 37.59% of the squared error and 17.71% of the absolute error; half of the squared error comes from 22 rows, half of the absolute error from 58 rows.
• When those 13 rows are trimmed, the ranking reverses on both measures (3.1186 against 3.1674 and 14.3617 against 14.6846); the whole advantage of the squared measure is in those rows.
• The coefficient of determination produces no third ranking; because its denominator is constant for all models, it is the form of the mean squared error read relative to the baseline and cannot be written without a baseline.

Next Step

All the numbers of this lesson were read from a single split: 756 rows of train, 252 rows of test. The gap between the two linear models came out as 0.0326 on the MAE and 0.0823 on the MSE. These are small numbers and so far there is no measure of fluctuation next to them — whether another split would give the same ranking is not known. The next lesson treats the split itself as a tuning choice: into how many folds the remaining data is split, how much the estimate fluctuates, how many trainings are done as the number of folds grows, and why the extreme that leaves a single row in each fold is not always better.