Confusion Matrix

The previous topic tried seven model families on the same held-out set and wrote next to every number how many candidates were tried to find it. The whole comparison ran on a single number: accuracy. The difference a family adds and the difference the tuning budget inflates were both read from this one number. Which errors it hides was never asked.

The test set has 61 suspicious subscribers out of 252. A model with an accuracy of 0.80 is wrong on fifty subscribers, but the direction of those fifty errors is not written: calling a non-suspicious subscriber suspicious and missing a suspicious one fall into the same cell. The two cannot stand in for each other — the cost of one is a wasted field visit, the cost of the other is an unnoticed illicit case.

• ET1. The set, the split and the features stay fixed throughout the course: the fictional subscriber table, the 756/252/252 split, the twelve features the Data Preparation and Feature Engineering course left behind. The only thing that changes is the model.
• ET2. The baseline does not change: telling every subscriber the most frequent class of the training set. This procedure gives 0.7579 on the test set.
• ET3. The positive class is the suspicious subscriber. The words positive and negative are not a value judgment; they are the name of the class being counted.
• ET4. The model produces a probability and in this lesson the decision threshold is fixed at 0.5. The threshold itself is left to the third lesson of this topic.
• ET5. Forty-nine candidates are tried: forty-two decision trees as the product of six depths and seven class weights, and a gradient-based linear model at seven class weights. The class weight is used in the form it had in the Data Preparation and Feature Engineering course.
• ET6. This lesson does not choose a model; the test set number of all forty-nine candidates is printed. Choosing from the validation set is the work of the third lesson.
• ET7. The cost has two items: one field visit for every flagged subscriber, one missed illicit case for every suspicious subscriber that is missed. The visit cost is written to all flagged subscribers; the outcome of the visit does not change the cost. The unit values are fictional and are not read from the data.

The Four Cells

A confusion matrix holds four numbers in a two-class decision. When the model says suspicious and the subscriber really is suspicious, it is a true positive; if not, a false positive. When the model does not say suspicious and the subscriber is suspicious, it is a false negative; if not, a true negative. Accuracy adds two of these four numbers and divides the sum by all four; the rest of the information is thrown away.

# evaluation.py — MODEL: the FICTIONAL subscriber table is rebuilt with the same seed.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, READINGS = [], {}
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": rating})
for k in SUBSCRIBERS:
    r = rng(SEED + 7000 + k["id"])
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()

DATA = []
for k in SUBSCRIBERS:
    v = READINGS.get(k["id"])
    if not v:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"])
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3]
         + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v),
         "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)}
    x["per_person"] = round(x["mean_consumption"] / x["household"], 3)
    for b in REGION:
        x["area_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)


def shuffle(data, seed):
    r, s = rng(seed), list(range(len(data)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [data[i] for i in s]


K = shuffle(DATA, SEED + 90000)
TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:]
FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period",
          "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west",
          "area_central"]


def impurity(rows):
    n = sum(x.get("w", 1.0) for x in rows)
    p = sum(x.get("w", 1.0) * x["suspicious"] for x in rows) / n
    return 2 * p * (1 - p)


def tree(rows, depth, fields, min_size=2):        # MODEL: weighted decision tree
    n = sum(x.get("w", 1.0) for x in rows)
    p = sum(x.get("w", 1.0) * x["suspicious"] for x in rows) / n
    best = None
    if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0:
        for f in fields:
            values = sorted({x[f] for x in rows})
            for v in (values[1:] if len(values) < 10 else
                      [values[int(i * len(values) / 10)] for i in range(1, 10)]):
                left = [x for x in rows if x[f] < v]
                right = [x for x in rows if x[f] >= v]
                if min(len(left), len(right)) < min_size:
                    continue
                gain = impurity(rows) - (sum(x.get("w", 1.0) for x in left) * impurity(left)
                                         + sum(x.get("w", 1.0) for x in right) * impurity(right)) / n
                if best is None or gain > best[0]:
                    best = (gain, f, v, left, right)
    if best is None or best[0] <= 1e-9:
        return {"p": p}
    return {"field": best[1], "threshold": best[2],
            "left": tree(best[3], depth - 1, fields, min_size),
            "right": tree(best[4], depth - 1, fields, min_size)}


def probability(node, x):
    while "p" not in node:
        node = node["left"] if x[node["field"]] < node["threshold"] else node["right"]
    return node["p"]


def weighted(train, w):
    return [dict(x, w=(w if x["suspicious"] else 1.0)) for x in train]


def matrix(score, data, threshold=0.5):           # tp, fp, fn, tn
    m = [0, 0, 0, 0]
    for x in data:
        i = (0 if x["suspicious"] else 1) + (0 if score(x) >= threshold else 2)
        m[i] += 1
    return m


N_SUSP = sum(x["suspicious"] for x in TEST)
print(f"test set {len(TEST)} subscribers, suspicious {N_SUSP}, "
      f"baseline {1 - N_SUSP / len(TEST):.4f}")
t6 = tree(TRAIN, 6, FIELDS)
m6 = matrix(lambda x: probability(t6, x), TEST)
print(f"depth 6 decision tree: accuracy {(m6[0] + m6[3]) / len(TEST):.4f}")
print(f"{'':>24}{'actual suspicious':>19}{'actual not suspicious':>23}")
print(f"{'called suspicious':>24}{m6[0]:>19}{m6[1]:>23}")
print(f"{'not called suspicious':>24}{m6[2]:>19}{m6[3]:>23}")

test set 252 subscribers, suspicious 61, baseline 0.7579
depth 6 decision tree: accuracy 0.7937
                          actual suspicious  actual not suspicious
       called suspicious                 24                     15
   not called suspicious                 37                    176

Accuracy is 0.7937 and above the baseline. The matrix makes the same model read differently: the model catches 24 of the 61 suspicious subscribers, misses 37, and flags 15 subscribers for nothing. Most of the accuracy comes from the bottom right cell — the majority in the set is already them. The whole 0.7579 of the baseline comes from that cell: the procedure that calls nobody suspicious collects 191 true negatives and accumulates 61 false negatives.

The Same Number, a Different Matrix

The way accuracy throws information away shows most clearly when two models give the same number. Forty-nine candidates are tried and the candidates that share the same accuracy value are searched for.

def scaler(train, fields):
    o = {a: sum(x[a] for x in train) / len(train) for a in fields}
    s = {a: (sum((x[a] - o[a]) ** 2 for x in train) / len(train)) ** 0.5 or 1.0 for a in fields}
    return lambda x: [(x[a] - o[a]) / s[a] for a in fields]


def linear(train, fields, w, epochs=300, step=0.5):   # MODEL: gradient-based linear model
    f, coef, b = scaler(train, fields), [0.0] * len(fields), 0.0
    X = [f(x) for x in train]
    y = [x["suspicious"] for x in train]
    sw = [(w if x["suspicious"] else 1.0) for x in train]
    n = sum(sw)
    for _ in range(epochs):
        gc, gb = [0.0] * len(fields), 0.0
        for i, xi in enumerate(X):
            z = b + sum(coef[j] * xi[j] for j in range(len(coef)))
            err = (1 / (1 + math.exp(-max(-30.0, min(30.0, z)))) - y[i]) * sw[i]
            gb += err
            for j in range(len(coef)):
                gc[j] += err * xi[j]
        b -= step * gb / n
        for j in range(len(coef)):
            coef[j] -= step * gc[j] / n

    def score(x):
        v = f(x)
        z = b + sum(coef[j] * v[j] for j in range(len(coef)))
        return 1 / (1 + math.exp(-max(-30.0, min(30.0, z))))
    return score


CANDIDATES = {}
for d in (3, 4, 5, 6, 7, 8):
    for w in (1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0):
        t = tree(weighted(TRAIN, w), d, FIELDS)
        CANDIDATES[f"tree d{d} w{w}"] = matrix(lambda x, t=t: probability(t, x), TEST)
for w in (1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0):
    s = linear(TRAIN, FIELDS, w)
    CANDIDATES[f"linear w{w}"] = matrix(s, TEST)

TIES = {}
for name, m in CANDIDATES.items():
    TIES.setdefault(round((m[0] + m[3]) / len(TEST), 4), []).append(name)
shared = {d: v for d, v in TIES.items() if len(v) > 1}
print(f"candidates tried {len(CANDIDATES)}, distinct accuracy values {len(TIES)}, "
      f"shared by more than one candidate {len(shared)}")
widest = max(shared.items(),
             key=lambda kv: max(CANDIDATES[a][0] for a in kv[1]) - min(CANDIDATES[a][0] for a in kv[1]))
print(f"\n{'candidate':<14}{'accuracy':>9}{'tp':>5}{'fp':>5}{'fn':>5}{'tn':>5}{'flagged':>10}")
for d in sorted(shared, reverse=True)[:3]:
    for name in shared[d]:
        m = CANDIDATES[name]
        print(f"{name:<14}{d:>9.4f}{m[0]:>5}{m[1]:>5}{m[2]:>5}{m[3]:>5}{m[0] + m[1]:>10}")
    print()
print(f"widest divergence at accuracy {widest[0]:.4f}: {' / '.join(widest[1])}")

candidates tried 49, distinct accuracy values 26, shared by more than one candidate 14

candidate      accuracy   tp   fp   fn   tn   flagged
tree d5 w1.0     0.8016   23   12   38  179        35
linear w1.0      0.8016   22   11   39  180        33
linear w1.5      0.8016   29   18   32  173        47

tree d3 w1.0     0.7937   17    8   44  183        25
tree d6 w1.0     0.7937   24   15   37  176        39

tree d4 w1.0     0.7817   16   10   45  181        26
tree d8 w1.0     0.7817   32   26   29  165        58

widest divergence at accuracy 0.7817: tree d4 w1.0 / tree d8 w1.0

The forty-nine candidates produce only twenty-six distinct accuracy values and fourteen of these are shared by more than one candidate. The first block of the table shows the highest accuracy in the set: three candidates give the value 0.8016 at once, and from two different model families at that. The numbers of suspicious subscribers they catch are 22, 23 and 29; the numbers of subscribers they flag are 33, 35 and 47. To someone looking by accuracy the three are the same model.

The last block is the widest divergence. Two candidates with an accuracy of 0.7817 catch 16 of the 61 suspicious subscribers in one and 32 in the other — exactly twice as many — and flag 58 subscribers against 26. The total number of correct predictions equalizes at 197 because the 16 true positives the second model gains swap exactly with the 16 true negatives it loses. Accuracy is indifferent to this swap; the matrix is not.

The Two Errors Do Not Cost the Same

The swap stops being a matter of indifference where the price of the two error types is counted. In the fictional set, one field visit is written for every flagged subscriber and one missed illicit case for every suspicious subscriber that is missed. The unit of the visit is taken as 1 and that of the missed illicit case as 6; these two numbers are not read from the data, they are given from outside.

VISIT_COST, MISS_COST = 1.0, 6.0


def cost(m):
    return VISIT_COST * (m[0] + m[1]) + MISS_COST * m[2]


print(f"{'procedure':<28}{'accuracy':>9}{'visits':>9}{'missed':>11}{'cost':>8}")
for name, m in (("baseline (none suspicious)", [0, 0, N_SUSP, len(TEST) - N_SUSP]),
                ("everyone suspicious", [N_SUSP, len(TEST) - N_SUSP, 0, 0]),
                ("tree d4 w1.0", CANDIDATES["tree d4 w1.0"]),
                ("tree d8 w1.0", CANDIDATES["tree d8 w1.0"]),
                ("tree d5 w1.0", CANDIDATES["tree d5 w1.0"]),
                ("linear w1.5", CANDIDATES["linear w1.5"])):
    print(f"{name:<28}{(m[0] + m[3]) / len(TEST):>9.4f}{m[0] + m[1]:>9}"
          f"{m[2]:>11}{cost(m):>8.0f}")

a, b = CANDIDATES["tree d4 w1.0"], CANDIDATES["tree d8 w1.0"]
ratio = ((a[0] + a[1]) - (b[0] + b[1])) / (b[2] - a[2])
print(f"\nbreak-even cost ratio {ratio:.2f}: the ranking flips once the cost of a missed "
      f"illicit case exceeds {ratio:.2f} times the cost of one field visit")
best_acc = max(CANDIDATES, key=lambda k: (CANDIDATES[k][0] + CANDIDATES[k][3]))
best_cost = min(CANDIDATES, key=lambda k: cost(CANDIDATES[k]))
print(f"best candidate by accuracy {best_acc}, cost {cost(CANDIDATES[best_acc]):.0f}")
print(f"best candidate by cost     {best_cost}, accuracy "
      f"{(CANDIDATES[best_cost][0] + CANDIDATES[best_cost][3]) / len(TEST):.4f}")
print(f"accuracy rank of the best candidate by cost "
      f"{sorted(CANDIDATES, key=lambda k: -(CANDIDATES[k][0] + CANDIDATES[k][3])).index(best_cost) + 1}"
      f" / {len(CANDIDATES)}")

procedure                    accuracy   visits     missed    cost
baseline (none suspicious)     0.7579        0         61     366
everyone suspicious            0.2421      252          0     252
tree d4 w1.0                   0.7817       26         45     296
tree d8 w1.0                   0.7817       58         29     232
tree d5 w1.0                   0.8016       35         38     263
linear w1.5                    0.8016       47         32     239

break-even cost ratio 2.00: the ranking flips once the cost of a missed illicit case exceeds 2.00 times the cost of one field visit
best candidate by accuracy tree d5 w1.0, cost 263
best candidate by cost     linear w3.5, accuracy 0.7460
accuracy rank of the best candidate by cost 20 / 49

The second row of the table shows on its own that accuracy is a one-directional measure. Calling everyone suspicious gives 0.2421 in accuracy — a number that looks unusable — but costs 252 and is cheaper than the baseline’s 366. Neither procedure has learned anything; one looks good on accuracy, the other on cost.

The cost of the two candidates with the same accuracy is 296 and 232. The 64 units between them are the price of the trade-off that accuracy hides. The break-even cost ratio comes out as 2.00: if the cost of a missed illicit case is below twice the cost of a field visit, the model that flags less is cheaper; if it is above, the model that flags more. This ratio cannot be read from the data; it comes from the operator of the measurement network — the same two numbers, two different decisions depending on which side of 2 the ratio falls.

The last two lines look at all forty-nine candidates. The best candidate by accuracy gives 263 in cost; the best candidate by cost is in twentieth place in the accuracy ranking. Being at the top on one metric does not bring being at the top on the other, and the difference is not at the level of chance. Accuracy is a cost calculation that counts the two errors equally; every calculation that does not count them equally picks a different candidate.

Summary

• The confusion matrix splits the two-class decision into four numbers — true positive, false positive, false negative, true negative — and accuracy adds two of the four and divides by the total number of observations.
• The forty-nine candidates produce only twenty-six distinct accuracy values; fourteen values are shared by more than one candidate, so the same number often corresponds to more than one error distribution.
• Two candidates with an accuracy of 0.7817 catch 16 and 32 of the 61 suspicious subscribers and flag 26 and 58 subscribers; because the true positives gained swap exactly with the true negatives lost, accuracy stays the same.
• When a field visit counts as 1 and a missed illicit case as 6, the cost of these two candidates is 296 and 232; the break-even cost ratio at which the ranking flips is 2.00, and this number comes not from the data but from the operation.
• Calling everyone suspicious stays far below the baseline in accuracy with 0.2421, while in cost it beats the baseline’s 366 with 252; the best candidate by accuracy and the best candidate by cost are not the same.

Next Step

This lesson counted the four cells but did not name the metrics derived from the cells. The imbalanced classes lesson of the Data Preparation and Feature Engineering course wrote that the share of the correctly flagged minority is called precision and the share of the caught minority is called recall, and left their definitions and how they read against each other to this course. The next lesson pays that debt: it derives four metrics from the four cells, ranks five models separately on these metrics and counts on how many metrics the leader of the ranking changes. Why the F1 score is the harmonic mean of two numbers and not their average is shown with numbers, together with the case in which the average gives a misleadingly high value.