Classification Metrics

The previous lesson split accuracy into four cells but did not name any of the metrics derived from the cells. The imbalanced classes lesson of the Data Preparation and Feature Engineering course had written two names — precision for the share of the minority that was flagged correctly, recall for the share of the minority that was caught — printed raw numbers, and left the definitions and how the two read against each other to this course. This lesson pays that debt.

The measure is this: five models are ranked separately on four metrics and it is counted on how many metrics the leader of the ranking changes. If the ranking is preserved when the metric changes, which one is chosen does not matter; if the order turns, the choice of metric is a decision that comes before the choice of model, and that decision cannot be read from the data.

• ET8. All four metrics are derived from a single confusion matrix; no new measurement is made, the same four numbers are combined in another way.
• ET9. The five models are taken from the forty-nine-candidate grid of the previous lesson. The number of candidates trained in this lesson is five; the selection was made for metric variety, not according to the number on the test set.
• ET10. The denominator of precision is the number of subscribers the model flags and it can be zero. When it is zero, precision is undefined, not zero; the baseline is in this situation.
• ET11. The denominator of recall is the 61 suspicious subscribers in the test set and is independent of the model.
• ET12. The F1 score is the harmonic mean of precision and recall; if the number of true positives is zero, it is written as zero.
• ET13. All four metrics are read at the decision threshold of 0.5. The number the threshold moves is measured in the next lesson.
• ET14. The rankings are built on the five models only; the baseline is a comparison row and does not enter the ranking.
• ET15. The numbers of the imbalanced classes lesson are not repeated. The set there had eleven features, the one here has twelve; the numbers cannot be compared one to one, the metric definitions are compared.

Four Metrics from Four Cells

Accuracy is the ratio of the sum of true positives and true negatives to all observations. Precision is how many of the subscribers called suspicious really are suspicious: the ratio of true positives to the sum of true positives and false positives. Recall is how many of the really suspicious subscribers were caught: the ratio of true positives to the sum of true positives and false negatives. Precision looks at the column the model flagged, recall at the suspicious row of the truth. The F1 score combines the two in a single number and the combination is not the arithmetic mean but the harmonic mean: $F_1 = \dfrac{2pr}{p+r}$, where $p$ is precision and $r$ is recall.

# evaluation.py — MODEL: the FICTIONAL subscriber table is rebuilt with the same seed.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, READINGS = [], {}
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": rating})
for k in SUBSCRIBERS:
    r = rng(SEED + 7000 + k["id"])
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()

DATA = []
for k in SUBSCRIBERS:
    v = READINGS.get(k["id"])
    if not v:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"])
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3]
         + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v),
         "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)}
    x["per_person"] = round(x["mean_consumption"] / x["household"], 3)
    for b in REGION:
        x["area_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)


def shuffle(data, seed):
    r, s = rng(seed), list(range(len(data)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [data[i] for i in s]


K = shuffle(DATA, SEED + 90000)
TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:]
FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period",
          "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west",
          "area_central"]


def impurity(rows):
    n = sum(x.get("w", 1.0) for x in rows)
    p = sum(x.get("w", 1.0) * x["suspicious"] for x in rows) / n
    return 2 * p * (1 - p)


def tree(rows, depth, fields, min_size=2):        # MODEL: weighted decision tree
    n = sum(x.get("w", 1.0) for x in rows)
    p = sum(x.get("w", 1.0) * x["suspicious"] for x in rows) / n
    best = None
    if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0:
        for f in fields:
            values = sorted({x[f] for x in rows})
            for v in (values[1:] if len(values) < 10 else
                      [values[int(i * len(values) / 10)] for i in range(1, 10)]):
                left = [x for x in rows if x[f] < v]
                right = [x for x in rows if x[f] >= v]
                if min(len(left), len(right)) < min_size:
                    continue
                gain = impurity(rows) - (sum(x.get("w", 1.0) for x in left) * impurity(left)
                                         + sum(x.get("w", 1.0) for x in right) * impurity(right)) / n
                if best is None or gain > best[0]:
                    best = (gain, f, v, left, right)
    if best is None or best[0] <= 1e-9:
        return {"p": p}
    return {"field": best[1], "threshold": best[2],
            "left": tree(best[3], depth - 1, fields, min_size),
            "right": tree(best[4], depth - 1, fields, min_size)}


def probability(node, x):
    while "p" not in node:
        node = node["left"] if x[node["field"]] < node["threshold"] else node["right"]
    return node["p"]


def weighted(train, w):
    return [dict(x, w=(w if x["suspicious"] else 1.0)) for x in train]


def matrix(score, data, threshold=0.5):           # tp, fp, fn, tn
    m = [0, 0, 0, 0]
    for x in data:
        i = (0 if x["suspicious"] else 1) + (0 if score(x) >= threshold else 2)
        m[i] += 1
    return m


def scaler(train, fields):
    o = {a: sum(x[a] for x in train) / len(train) for a in fields}
    s = {a: (sum((x[a] - o[a]) ** 2 for x in train) / len(train)) ** 0.5 or 1.0 for a in fields}
    return lambda x: [(x[a] - o[a]) / s[a] for a in fields]


def linear(train, fields, w, epochs=300, step=0.5):   # MODEL: gradient-based linear model
    f, coef, b = scaler(train, fields), [0.0] * len(fields), 0.0
    X = [f(x) for x in train]
    y = [x["suspicious"] for x in train]
    sw = [(w if x["suspicious"] else 1.0) for x in train]
    n = sum(sw)
    for _ in range(epochs):
        gc, gb = [0.0] * len(fields), 0.0
        for i, xi in enumerate(X):
            z = b + sum(coef[j] * xi[j] for j in range(len(coef)))
            err = (1 / (1 + math.exp(-max(-30.0, min(30.0, z)))) - y[i]) * sw[i]
            gb += err
            for j in range(len(coef)):
                gc[j] += err * xi[j]
        b -= step * gb / n
        for j in range(len(coef)):
            coef[j] -= step * gc[j] / n

    def score(x):
        v = f(x)
        z = b + sum(coef[j] * v[j] for j in range(len(coef)))
        return 1 / (1 + math.exp(-max(-30.0, min(30.0, z))))
    return score


N_SUSP = sum(x["suspicious"] for x in TEST)
MODEL = {}
for name, d, w in (("depth 5 tree", 5, 1.0), ("depth 3 tree", 3, 1.0),
                   ("depth 8 tree", 8, 1.0), ("depth 6 tree w3", 6, 3.0)):
    t = tree(weighted(TRAIN, w), d, FIELDS)
    MODEL[name] = matrix(lambda x, t=t: probability(t, x), TEST)
MODEL["linear w2.5"] = matrix(linear(TRAIN, FIELDS, 2.5), TEST)
MODEL["baseline"] = [0, 0, N_SUSP, len(TEST) - N_SUSP]


def metrics(m):
    tp, fp, fn, tn = m
    p = tp / (tp + fp) if tp + fp else float("nan")
    r = tp / (tp + fn)
    return (tp + tn) / sum(m), p, r, (2 * p * r / (p + r) if tp else 0.0)


print(f"{'model':<18}{'accuracy':>9}{'precision':>10}{'recall':>11}{'F1':>8}"
      f"{'flagged':>9}")
for name, m in MODEL.items():
    a, p, r, f = metrics(m)
    print(f"{name:<18}{a:>9.4f}{('undefined' if p != p else f'{p:.4f}'):>10}"
          f"{r:>11.4f}{f:>8.4f}{m[0] + m[1]:>9}")

model              accuracy precision     recall      F1  flagged
depth 5 tree         0.8016    0.6571     0.3770  0.4792       35
depth 3 tree         0.7937    0.6800     0.2787  0.3953       25
depth 8 tree         0.7817    0.5517     0.5246  0.5378       58
depth 6 tree w3      0.7302    0.4545     0.5738  0.5072       77
linear w2.5          0.7659    0.5135     0.6230  0.5630       74
baseline             0.7579 undefined     0.0000  0.0000        0

The baseline row breaks three metrics at once. The precision of a procedure that flags no subscriber is undefined — there is nothing to divide — and its recall and F1 score are zero. The same procedure that reads 0.7579 on accuracy gives an unusable result on the other three metrics. This is the defined form of the table the imbalanced classes lesson showed with raw numbers.

In the five-model rows two columns move in opposite directions. As the number of flagged subscribers rises from 25 to 77, precision falls from 0.6800 to 0.4545 and recall rises from 0.2787 to 0.5738. The model that flags less is right more often when it flags, but misses most; the model that flags more misses fewer, but more than half of what it flags is wasted. The accuracy column is almost indifferent to this trade-off: there are 0.0714 points between the highest and the lowest, 0.2255 points in precision, 0.3443 points in recall.

The Ranking Turns with the Metric

The same five models are ranked separately on four metrics and the rankings are printed side by side.

FIVE = [a for a in MODEL if a != "baseline"]
METRICS = ("accuracy", "precision", "recall", "F1")
RANK = {o: {a: j + 1 for j, a in enumerate(
    sorted(FIVE, key=lambda a: -metrics(MODEL[a])[i]))}
        for i, o in enumerate(METRICS)}
print(f"{'model':<18}" + "".join(f"{o:>11}" for o in METRICS))
for a in FIVE:
    print(f"{a:<18}" + "".join(f"{RANK[o][a]:>11}" for o in METRICS))
leaders = [min(RANK[o], key=RANK[o].get) for o in METRICS]
print(f"\nmodel leading each metric: {' | '.join(leaders)}")
print(f"{len(set(leaders))} distinct models lead on the four metrics; the model chosen by "
      f"accuracy is not in the lead on {sum(1 for b in leaders[1:] if b != leaders[0])} metrics")
flips = sum(1 for i in range(len(FIVE)) for j in range(i + 1, len(FIVE))
            if (RANK["accuracy"][FIVE[i]] - RANK["accuracy"][FIVE[j]])
            * (RANK["F1"][FIVE[i]] - RANK["F1"][FIVE[j]]) < 0)
print(f"model pairs that swap places between accuracy and F1 {flips} / "
      f"{len(FIVE) * (len(FIVE) - 1) // 2}")

model                accuracy  precision     recall         F1
depth 5 tree                1          2          4          4
depth 3 tree                2          1          5          5
depth 8 tree                3          3          3          2
depth 6 tree w3             5          5          2          3
linear w2.5                 4          4          1          1

model leading each metric: depth 5 tree | depth 3 tree | linear w2.5 | linear w2.5
3 distinct models lead on the four metrics; the model chosen by accuracy is not in the lead on 3 metrics
model pairs that swap places between accuracy and F1 7 / 10

The accuracy column descends smoothly from 1 to 5 because the models were listed by accuracy. The other three columns do not recognize that order. The depth 5 tree, first in accuracy, is second in precision and fourth in recall and the F1 score. The depth 3 tree, second in accuracy, is first in precision but last in recall and the F1 score. The linear model, fourth in accuracy, is first in recall and the F1 score.

The numbers gather in three places. Three different models take the lead on the four metrics; the model chosen by accuracy is not in the lead on three of the four metrics; seven of the ten model pairs swap places between accuracy and F1. The ranking is built not on the measure but on the metric. Writing which budget a model’s number was found with is not enough — it is also necessary to write which metric it was ranked by, because five models have entered four separate races on four metrics.

Why the Harmonic Mean Is Not the Mean

If the F1 score combined precision and recall with the arithmetic mean, a procedure with one side high and the other near zero would look to be in the middle. The harmonic mean gives weight to the smaller one. The difference grows as the two numbers move apart, and the growth can be counted.

print(f"{'precision':>9}{'recall':>11}{'mean':>10}{'F1':>8}{'diff':>8}")
for p, r in ((1.00, 1 / 61), (0.90, 0.10), (0.70, 0.30), (0.60, 0.40), (0.50, 0.50)):
    o, f = (p + r) / 2, 2 * p * r / (p + r)
    print(f"{p:>9.4f}{r:>11.4f}{o:>10.4f}{f:>8.4f}{o - f:>8.4f}")

MEAN = {a: sum(metrics(MODEL[a])[1:3]) / 2 for a in FIVE}
print(f"\n{'model':<18}{'mean':>10}{'F1':>8}{'diff':>8}{'mean/F1 rank':>15}")
for j, a in enumerate(sorted(FIVE, key=lambda a: -MEAN[a])):
    f = metrics(MODEL[a])[3]
    ranks = str(j + 1) + " / " + str(RANK["F1"][a])
    print(f"{a:<18}{MEAN[a]:>10.4f}{f:>8.4f}{MEAN[a] - f:>8.4f}{ranks:>15}")

precision     recall      mean      F1    diff
   1.0000     0.0164    0.5082  0.0323  0.4759
   0.9000     0.1000    0.5000  0.1800  0.3200
   0.7000     0.3000    0.5000  0.4200  0.0800
   0.6000     0.4000    0.5000  0.4800  0.0200
   0.5000     0.5000    0.5000  0.5000  0.0000

model                   mean      F1    diff   mean/F1 rank
linear w2.5           0.5682  0.5630  0.0053          1 / 1
depth 8 tree          0.5382  0.5378  0.0003          2 / 2
depth 5 tree          0.5171  0.4792  0.0379          3 / 4
depth 6 tree w3       0.5142  0.5072  0.0069          4 / 3
depth 3 tree          0.4793  0.3953  0.0840          5 / 5

The first row of the first table shows the extreme case: a procedure that flags a single subscriber and is right on that subscriber has a precision of 1.0000 and a recall of 0.0164. The arithmetic mean gives 0.5082 — a number higher than all five real models — and the F1 score gives 0.0323. The difference is 0.4759. The three middle rows meet the same arithmetic mean, 0.5000, with three different F1 values: 0.1800, 0.4200 and 0.4800. When the two numbers are equal the difference goes to zero. The only thing the harmonic mean does is make the combined number collapse too when one of the two metrics collapses.

The second table shows the same calculation on real models. In the depth 8 tree, where precision and recall are close to each other, the difference between the two means is 0.0003; in the depth 3 tree, where the two diverge most, it is 0.0840. The ranking also breaks somewhere: the depth 5 tree, third by the arithmetic mean, is fourth on the F1 score, and the weighted tree, fourth, moves to third place. The sum of precision and recall of the two models is almost the same (0.5171 and 0.5142) but its distribution is different, and the harmonic mean sees the distribution.

Summary

• Precision is read from the column the model flagged, recall from the suspicious row of the truth; accuracy uses neither of them, it adds the two diagonals of the four cells.
• The precision of the baseline is undefined because no subscriber was flagged; the same procedure reads 0.7579 on accuracy while giving zero on recall and the F1 score.
• The five models are squeezed between 0.7302 and 0.8016 on accuracy while they spread 0.2255 points on precision and 0.3443 points on recall; accuracy is almost indifferent to this trade-off.
• Three different models take the lead on the four metrics, the model chosen by accuracy is not in the lead on three metrics, and seven of the ten model pairs swap places between accuracy and F1.
• The F1 score is the harmonic mean: for a procedure with a precision of 1.0000 and a recall of 0.0164 the arithmetic mean gives 0.5082 and the F1 score 0.0323; when the two numbers are equal, the two means are equal too.

Next Step

All four metrics of this lesson were read at the decision threshold of 0.5 and that threshold was never questioned. Yet the number the models produce is a probability; 0.5 is a choice that comes from outside, and it changes the table from start to finish. The next lesson sweeps the threshold from 0 to 1, prints precision and recall side by side at every threshold and chooses the operating point with the cost calculation of the first lesson. The same sweep produces two curves — the ROC curve and the precision–recall curve — and which of them misleads on an imbalanced set, and how far apart the areas under the two curves diverge on the same model, is shown by counting.