Missing Value Strategies

In the previous three lessons the satisfaction column was assumed to be filled for every subscriber. In the raw record it is not so. Satisfaction is a survey response, and there are subscribers who did not answer the survey. What to do about the gap is a decision, and until now that decision was never visible, because it was taken silently.

The imputation procedure itself is not built in this lesson. Finding the missing value, imputing it with the median and dropping the row were measured in the Data Collection and Preparation course of the Data Analytics curriculum; the measure there was how many rows were affected. The question here is different: how much do the same three options move the number on the held-out set, and is the missingness itself a feature.

• PS26. Not answering the survey is fictional and is not independent of the label: subscribers suspected of illicit use or a fault answer the survey less often. It was put into the setup deliberately, and it is what this lesson measures.
• PS27. Missingness is only in the satisfaction column. The other ten columns and the label are complete.
• PS28. The imputation value is a fitted object: the median is computed only from the filled rows in the training set, and that value is applied to both sets.
• PS29. The imputation statistic is itself a decision: the median was chosen because satisfaction is measured on an ordinal scale, and on that scale the mean is not a legitimate summary. If the choice changes, the imputed value changes too.
• PS30. The missingness indicator is a binary column that says whether that column is empty for the row. It is added next to the imputation, not in its place.
• PS31. The measurement set, the split, the model and the baseline are the same as in the earlier lessons: 756/252/252, a depth 6 decision tree, 0.7579 on the test set.
• PS32. The number measured with the complete column is printed as an upper bound: what would be read if there had been no missingness at all. This row is not a strategy but a comparison line.
• PS33. The test set is 252 subscribers, and one subscriber is worth 0.0040.

How Much the Gap Itself Says

The setup produces the table, puts the survey missingness in, and prints the relation of missingness to the label in both sets.

# missing.py — MODEL. The fictional subscriber table is regenerated with the same seed; the
# satisfaction column is made incomplete as a FICTIONAL survey response, as in the raw record.
import math

SEED, CANDIDATES, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, READINGS = [], {}
for i in range(CANDIDATES):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    satisfaction = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": satisfaction})
for k in SUBSCRIBERS:
    r = rng(SEED + 7000 + k["id"])
    for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - d * 0.05) * 100 + 0.5) / 100)
        r()

DATA = []
for k in SUBSCRIBERS:
    v = READINGS.get(k["id"])
    if v is None:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"])
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3]
         + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v),
         "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)}
    for b in REGION:
        x["b_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)

survey_rng = rng(SEED + 31000)              # not answering the survey is FICTIONAL
for x in DATA:
    x["survey"] = None if survey_rng() < (0.55 if x["suspicious"] else 0.15) else x["satisfaction"]


def shuffle(data, seed):
    r, s = rng(seed), list(range(len(data)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [data[i] for i in s]


K = shuffle(DATA, SEED + 90000)
TRAIN, TEST = K[:756], K[1008:]
BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST)
FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings",
          "b_north", "b_south", "b_east", "b_west", "b_central"]
print(f"{'set':<8} {'rows':>6} {'missing':>7} {'missing rate':>12} "
      f"{'suspicious, missing':>19} {'complete':>9}")
for name, s in (("train", TRAIN), ("test", TEST)):
    e = [x for x in s if x["survey"] is None]
    t = [x for x in s if x["survey"] is not None]
    print(f"{name:<8} {len(s):>6} {len(e):>7} {len(e) / len(s):>12.4f} "
          f"{sum(x['suspicious'] for x in e) / len(e):>19.4f} "
          f"{sum(x['suspicious'] for x in t) / len(t):>9.4f}")
d = sorted(x["survey"] for x in TRAIN if x["survey"] is not None)
MEDIAN = d[len(d) // 2]                     # FIT: only from the training set
print(f"median fitted on train {MEDIAN}, baseline {BASELINE:.4f}")

set        rows missing missing rate suspicious, missing  complete
train       756     163       0.2156              0.5276    0.1383
test        252      58       0.2302              0.6207    0.1289
median fitted on train 4, baseline 0.7579

The last two columns are the whole subject of the lesson. The suspicious rate is 0.5276 among subscribers who did not answer the survey and 0.1383 among those who did. The gap is not empty; it says something other than the value the column carries. The imputation decision either keeps this difference or erases it.

The imputation value is itself a fitted object. The median of the filled rows in the training set comes out as 4, and this single number is applied to both sets. Looking at the median of the test set itself would have produced a more “correct” imputation, but computing that value would be looking at the held-out set; the discipline set up in scaling applies here exactly the same. It is also expected that the missingness rate does not come out the same in the two sets: 0.2156 in train, 0.2302 in test, and the difference comes from the randomness of the split.

Three Strategies, Three Numbers

The code below writes the model and measures five strategies together with an upper bound row.

def gini(s):
    p = sum(x["suspicious"] for x in s) / len(s) if s else 0.0
    return 2 * p * (1 - p)


def tree(s, depth, fields, min_size=2):     # MODEL: decision tree, depth 6
    p = sum(x["suspicious"] for x in s) / len(s)
    best = None
    if depth and len(s) >= 2 * min_size and 0.0 < p < 1.0:
        for a in fields:
            d = sorted({x[a] for x in s})
            for v in (d[1:] if len(d) < 10 else
                      [d[int(i * len(d) / 10)] for i in range(1, 10)]):
                left = [x for x in s if x[a] < v]
                right = [x for x in s if x[a] >= v]
                if min(len(left), len(right)) < min_size:
                    continue
                k = gini(s) - (len(left) * gini(left) + len(right) * gini(right)) / len(s)
                if best is None or k > best[0]:
                    best = (k, a, v, left, right)
    if best is None or best[0] <= 1e-9:
        return {"prediction": int(p > 0.5)}
    return {"field": best[1], "threshold": best[2],
            "left": tree(best[3], depth - 1, fields, min_size),
            "right": tree(best[4], depth - 1, fields, min_size)}


def predict(d, x):
    while "prediction" not in d:
        d = d["left"] if x[d["field"]] < d["threshold"] else d["right"]
    return d["prediction"]


def impute(s, value):                       # APPLY: a single value for the gap
    return [dict(x, satisfaction=(value if x["survey"] is None else x["survey"]))
            for x in s]


def add_indicator(s, value):                # APPLY: imputation + missingness column
    return [dict(x, satisfaction=(value if x["survey"] is None else x["survey"]),
                 survey_missing=int(x["survey"] is None)) for x in s]


RESULT = {}


def measure(name, fields, e, s):
    m = tree(e, 6, fields)
    train_acc = sum(predict(m, x) == x["suspicious"] for x in e) / len(e)
    test_acc = sum(predict(m, x) == x["suspicious"] for x in s) / len(s)
    RESULT[name] = (len(e), len(fields), train_acc, test_acc)
    print(f"{name:<35} {len(e):>7} {len(fields):>7} {train_acc:>8.4f} {test_acc:>8.4f}")


IMPUTED_TRAIN, IMPUTED_TEST = impute(TRAIN, MEDIAN), impute(TEST, MEDIAN)
print(f"{'procedure':<35} {'rows':>7} {'columns':>7} {'train':>8} {'test':>8}")
measure("column dropped entirely", [a for a in FIELDS if a != "satisfaction"], TRAIN, TEST)
measure("imputed with median", FIELDS, IMPUTED_TRAIN, IMPUTED_TEST)
measure("imputation + missingness indicator", FIELDS + ["survey_missing"],
        add_indicator(TRAIN, MEDIAN), add_indicator(TEST, MEDIAN))
measure("imputed with constant -1", FIELDS, impute(TRAIN, -1), impute(TEST, -1))
measure("missing rows dropped from train", FIELDS,
        [x for x in IMPUTED_TRAIN if x["survey"] is not None], IMPUTED_TEST)
measure("complete column (upper bound)", FIELDS, TRAIN, TEST)

procedure                              rows columns    train     test
column dropped entirely                 756      10   0.8439   0.7540
imputed with median                     756      11   0.8466   0.7619
imputation + missingness indicator      756      12   0.8823   0.8333
imputed with constant -1                756      11   0.8823   0.8333
missing rows dropped from train         593      11   0.9191   0.7659
complete column (upper bound)           756      11   0.8466   0.7778

Median imputation gives 0.7619: only one subscriber above the baseline. Dropping the column entirely gives 0.7540: below the baseline. Between the two there is a difference of two subscribers, and this is below the measurement resolution. The two common options cannot be told apart in this table.

The train column should be read separately, because it divides the strategies into two groups. Imputation and column dropping stay around 0.84 on train; the indicator and the constant value rise to 0.8823; row dropping rises to 0.9191. The rise of the first two on train finds its counterpart on the held-out set, that of the third does not. What kind of increase it is on the training set can be understood only when the held-out set number is written next to it; this is the form in which the rule of the previous course works in this table.

The Contribution Comes from the Gap

Reading the differences against the imputation row separates which decision is really a step.

t = RESULT["imputed with median"]
print(f"{'procedure':<35} {'contribution':>12} {'subscribers':>11}")
for name in ("column dropped entirely", "imputation + missingness indicator",
             "imputed with constant -1", "missing rows dropped from train",
             "complete column (upper bound)"):
    k = RESULT[name][3] - t[3]
    print(f"{name:<35} {k:>+12.4f} {round(k * 252):>11}")
ae = [x for x in TRAIN if x["survey"] is None]
print(f"\nof the {len(ae)} rows dropped from train, {sum(x['suspicious'] for x in ae)} are "
      f"suspicious; suspicious left in train "
      f"{sum(x['suspicious'] for x in TRAIN) - sum(x['suspicious'] for x in ae)} / "
      f"{sum(x['suspicious'] for x in TRAIN)}")
print(f"in test, a prediction must still be produced for the "
      f"{sum(1 for x in TEST if x['survey'] is None)} subscribers whose survey is missing; "
      f"row dropping cannot be applied on the held-out set")

procedure                           contribution subscribers
column dropped entirely                  -0.0079          -2
imputation + missingness indicator       +0.0714          18
imputed with constant -1                 +0.0714          18
missing rows dropped from train          +0.0040           1
complete column (upper bound)            +0.0159           4

of the 163 rows dropped from train, 86 are suspicious; suspicious left in train 82 / 168
in test, a prediction must still be produced for the 58 subscribers whose survey is missing; row dropping cannot be applied on the held-out set

One row is worth reading: the missingness indicator is +0.0714, that is, eighteen subscribers. This is the largest contribution of the course so far, and where it comes from is clear. While imputation closes the gap, it also erases which rows were empty; the indicator puts that information back. In a table where the suspicious rate among subscribers who did not answer the survey is 0.5276, “not having answered” says more in itself than the satisfaction score.

The proof of this is in the upper bound row. Without any missingness the model would read 0.7778; the indicator exceeds it by 0.0555. If a feature gives a better number than its complete form, the contribution comes not from the value of the column but from the way the column is missing.

Imputation with a constant value gives the same number, 0.8333. The reason is the way the tree works: since the −1 given to the missing rows is a single value outside the scale, the tree can separate it with a single threshold and turns the column into an implicit indicator. This equality has been measured here for the tree; for a method that computes distance by summing the columns or multiplies them by a weight, −1 is an ordinary number, and its result should be measured separately.

The row dropping line carries two separate problems. It reads 0.9191 on train but stays at 0.7659 on test; on a set reduced to 593 rows the tree memorizes more. The real problem is in the second output: 86 of the 163 rows dropped are suspicious, that is, more than half of the minority class is dropped, and of the 168 suspicious in the training set only 82 remain. The third and decisive obstacle is this: there are 58 subscribers in the test set whose survey is missing, and a prediction has to be produced for them too. Dropping rows is an option on training, it is not an option on the held-out set; this is why it can never be a strategy on its own.

The source of the contribution must also be asked. Not answering the survey is a fact known at prediction time and, as it stands, a real feature; the contribution comes from learning. But if the missingness arises as a result of the inspection — if a subscriber found suspicious stops answering the survey — then the same column turns into a feature that sees the future and the contribution comes from leakage. The two cases are identical in the table. The distinction is made only by knowing when the missingness arises, and that information is not in the data.

Summary

• The suspicious rate is 0.5276 among subscribers who did not answer the survey and 0.1383 among those who did; the missingness is related to the label, and imputation erases this relation.
• Median imputation gives 0.7619 and dropping the column entirely gives 0.7540; the two-subscriber difference between them is below the measurement resolution, and the two options cannot be told apart.
• Adding a missingness indicator next to the imputation gives 0.8333, a contribution of +0.0714, that is, eighteen subscribers; this number also exceeds by 0.0555 the upper bound of 0.7778 that would be read if there had been no missingness at all.
• Imputation with a constant value gives the same number as the indicator on the tree, because a single value outside the scale is an implicit indicator for the tree; this equality has not been measured for other model families.
• Dropping missing rows from training removes more than half of the minority class (82 of 168 suspicious remain) and cannot be applied on the held-out set at all, because a prediction is needed for the 58 subscribers in the test set.

Next Step

In this lesson the dropping of half of the minority class was seen as a side effect, but the size difference between the classes itself was never measured. In the training set 168 of 756 subscribers are suspicious, and in the test set 61 of 252. At this rate accuracy is a misleading measure: the baseline that marks no one as suspicious reads 0.7579, and the number of suspicious subscribers it catches is zero. The next lesson prints these two numbers side by side and measures the three answers given to the imbalance — oversampling the minority, undersampling the majority and class weight — both in accuracy and in the number of suspicious subscribers caught. The theory of measures and the definitions of precision and recall are not the subject of this course and are left to the Supervised Learning course.