Time and Text Features

The previous lesson generated six features, measured all six one by one, and left a single column behind: consumption per person, the ratio feature that carried the line before the decision from 0.7778 to 0.7897. The other five were removed from the set. From this lesson on, the line before the decision is the line of twelve columns.

The subscriber table has two more fields that have not been given to the model, and neither is a number: the date on which the contract started and the short note the field crew left during the reading. A decision tree cannot split either of them as they are; a date is an ordered value, but its ordering does not carry the seasonality of the year, and a text is not ordered at all. Several numbers can be extracted from each of them, and this is exactly the measure of this lesson: how many numbers a field yields and what the total contribution of those numbers is.

• FS8. The line before the decision is twelve columns: the eleven columns of the Introduction to Machine Learning course and per_person. The model does not change: a decision tree of depth 6. The baseline is 0.7579.
• FS9. The contract date and the field note are fictional; they are generated from the same seed (20260218) depending on the subscriber id and added to the table in this lesson.
• FS10. The field note is written during the reading period; the suspicion label becomes final after the period closes. The content of the note is at hand at prediction time.
• FS11. The seal trace record enters the note with probability 0.42 for suspicious subscribers and 0.10 for the others; the access record enters with probability 0.30 if the period count is below three and 0.08 otherwise.
• FS12. The extra note enters with probability 0.25 and is one of three equally likely topics; the vehicle record thus falls into the note with a probability of about 0.08 and is independent of the label; it is a trap indicator.
• FS13. String operations were built in the data frames topic of the Data Analysis with Python course; the string routine is not repeated here, only the number extracted from the text is measured. The bag of words and the embedding space are not the subject of this course.
• FS14. Contribution is the difference of the twelve columns’ number on the held-out set from the line before the decision; one row is 0.0040.

Five Numbers from a Date, Five from a Note

The setup produces the same set as the previous lessons; the only difference is that the per_person column, whose persistence has been measured, is added from the start.

# COMMON — MODEL: the FICTIONAL subscriber table of the Introduction to Machine Learning course is rebuilt with the same seed;
# the model is a decision tree of depth 6. The only new column left from the previous lesson is per_person.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, DATA = [], []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": rating})
for k in SUBSCRIBERS:
    r, v = rng(SEED + 7000 + k["id"]), []
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        v.append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()
    if not v:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"])
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3]
         + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"],
         "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35),
         "region_name": k["region"][0]}
    for b in REGION:
        x["area_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)

BASE = ["mean_consumption", "volatility", "household", "satisfaction", "period",
        "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"]
BASE_FIELDS = BASE + ["per_person"]
for x in DATA:
    x["per_person"] = round(x["mean_consumption"] / x["household"], 3)


def shuffle(data, seed):
    r, s = rng(seed), list(range(len(data)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [data[i] for i in s]


K = shuffle(DATA, SEED + 90000)
TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:]


def gini(rows):
    p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0
    return 2 * p * (1 - p)


def tree(rows, depth, fields, min_size=2):
    p, best = sum(x["suspicious"] for x in rows) / len(rows), None
    if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0:
        for f in fields:
            values = sorted({x[f] for x in rows})
            for v in (values[1:] if len(values) < 10 else
                      [values[int(i * len(values) / 10)] for i in range(1, 10)]):
                left = [x for x in rows if x[f] < v]
                right = [x for x in rows if x[f] >= v]
                if min(len(left), len(right)) < min_size:
                    continue
                k = gini(rows) - (len(left) * gini(left) + len(right) * gini(right)) / len(rows)
                if best is None or k > best[0]:
                    best = (k, f, v, left, right)
    if best is None or best[0] <= 1e-9:
        return {"prediction": int(p > 0.5)}
    return {"field": best[1], "threshold": best[2],
            "left": tree(best[3], depth - 1, fields, min_size),
            "right": tree(best[4], depth - 1, fields, min_size)}


def predict(node, x):
    while "prediction" not in node:
        node = node["left"] if x[node["field"]] < node["threshold"] else node["right"]
    return node["prediction"]


def accuracy(node, rows):
    return sum(predict(node, x) == x["suspicious"] for x in rows) / len(rows)


def baseline(train, rows):
    c = int(sum(x["suspicious"] for x in train) * 2 > len(train))
    return sum(x["suspicious"] == c for x in rows) / len(rows)


def measure(fields, depth=6):
    m = tree(TRAIN, depth, fields)
    return accuracy(m, TRAIN), accuracy(m, TEST)

The five numbers extracted from the date correspond to two different questions. The month, the quarter, the day of the week and the day of the year are seasonal indicators: each is a number that cycles within itself, and their purpose is to catch a seasonal pattern. The month and the quarter give the same value on the same calendar date every year; the day of the week shifts from year to year, and the day of the year moves by one day in a leap year. The contract age, on the other hand, is a linear number and grows every day. The five numbers extracted from the note also fall into two groups: two of them measure the form of the text (character count, word count), and three show the presence of a particular keyword.

import datetime

START = datetime.date(2024, 1, 1)
CORE = ["meter access completed", "reading okay", "subscriber was out",
        "dirty cover", "readings repeated", "valve was inspected"]
EXTRA = ["car parked nearby", "doorbell not working", "garden gates open"]

for x in DATA:
    r = rng(SEED + 61000 + x["id"])
    x["contract_day"] = int(r() * 1460)
    g = START + datetime.timedelta(days=x["contract_day"])
    x["month"], x["quarter"] = g.month, (g.month - 1) // 3 + 1
    x["weekday"], x["day_of_year"] = g.weekday(), g.timetuple().tm_yday
    x["contract_age"] = 1460 - x["contract_day"]
    p = [CORE[int(r() * len(CORE))]]
    if r() < (0.42 if x["suspicious"] else 0.10):
        p.append("seal was torn")
    if r() < (0.30 if x["period"] < 3 else 0.08):
        p.append("access unavailable")
    if r() < 0.25:
        p.append(EXTRA[int(r() * len(EXTRA))])
    n = ", ".join(p)
    x["field_note"], x["note_length"], x["note_words"] = n, len(n), len(n.split())
    x["kw_seal"] = int("seal" in n)
    x["kw_access"] = int("access unavailable" in n)
    x["kw_car"] = int("car" in n)

TIME = ["month", "quarter", "weekday", "day_of_year", "contract_age"]
TEXT = ["note_length", "note_words", "kw_seal", "kw_access", "kw_car"]
for i in (4, 9, 10):
    print(f"note example: {DATA[i]['field_note']}")
print(f"seal trace recorded in {sum(x['kw_seal'] for x in DATA)} rows; in "
      f"{sum(x['kw_seal'] for x in DATA if x['suspicious']) / sum(x['suspicious'] for x in DATA):.3f}"
      f" of the suspicious, in "
      f"{sum(x['kw_seal'] for x in DATA if not x['suspicious']) / sum(1 - x['suspicious'] for x in DATA):.3f}"
      f" of the others\n")
BEFORE = measure(BASE_FIELDS)
print(f"{'block':<16} {'columns':>7} {'train':>9} {'held_out':>9} {'contrib':>8}")
print(f"{'-- none --':<16} {0:>7} {BEFORE[0]:>9.4f} {BEFORE[1]:>9.4f}")
for name, fs in (("from date", TIME), ("from text", TEXT), ("both", TIME + TEXT)):
    e, s = measure(BASE_FIELDS + fs)
    print(f"{name:<16} {len(fs):>7} {e:>9.4f} {s:>9.4f} {s - BEFORE[1]:>+8.4f}")

note example: readings repeated
note example: subscriber was out, access unavailable, car parked nearby
note example: dirty cover, seal was torn
seal trace recorded in 219 rows; in 0.453 of the suspicious, in 0.096 of the others

block            columns     train  held_out  contrib
-- none --             0    0.8519    0.7897
from date              5    0.8611    0.7698  -0.0198
from text              5    0.8942    0.8214  +0.0317
both                  10    0.8955    0.8095  +0.0198

Side by side, the three numbers show that there is no link between the number of fields and the contribution. Five numbers were extracted from each of two fields; one earned 0.0317, the other lost 0.0198. When ten numbers are given together, the total contribution falls to 0.0198 — that is, the date field takes back three of the eight subscribers the text field earned. The training number, on the other hand, rises in all three setups: from 0.8519 to 0.8955. That the training number rises as the column count grows is a requirement of the definition, not a measurement.

That the date block comes out negative is not surprising, because in this fiction no link has been built between the contract date and the suspicion. The cause of the loss is not the absence of a link but five columns competing as split candidates: the day-of-year column has 366 distinct values, and among that many threshold candidates it is easy to find one that looks good on the training set. A noise column with many values is more dangerous than a noise column with few.

Where the Contribution Comes From

When the blocks are opened one by one, it becomes visible which column the total numbers come from.

print(f"{'single feature':<16} {'source':<8} {'train':>9} {'held_out':>9} {'contrib':>8}")
for f in TIME + TEXT:
    e, s = measure(BASE_FIELDS + [f])
    print(f"{f:<16} {'date' if f in TIME else 'text':<8} {e:>9.4f} {s:>9.4f} "
          f"{s - BEFORE[1]:>+8.4f}")
for name, fs in (("only three keywords", ["kw_seal", "kw_access", "kw_car"]),
                 ("only two counters", ["note_length", "note_words"]),
                 ("only the seal", ["kw_seal"])):
    e, s = measure(BASE_FIELDS + fs)
    print(f"\n{name:<20} {e:.4f} {s:.4f} {s - BEFORE[1]:+.4f}", end="")
print()

single feature   source       train  held_out  contrib
month            date        0.8519    0.7897  +0.0000
quarter          date        0.8519    0.7897  +0.0000
weekday          date        0.8532    0.7778  -0.0119
day_of_year      date        0.8519    0.7897  +0.0000
contract_age     date        0.8585    0.7817  -0.0079
note_length      text        0.8823    0.7540  -0.0357
note_words       text        0.8783    0.7381  -0.0516
kw_seal          text        0.8876    0.8254  +0.0357
kw_access        text        0.8519    0.7857  -0.0040
kw_car           text        0.8519    0.7897  +0.0000

only three keywords  0.8876 0.8254 +0.0357
only two counters    0.8942 0.7381 -0.0516
only the seal        0.8876 0.8254 +0.0357

Nine of the ten features give zero or negative contribution. The whole gain comes from the kw_seal column: 0.0357 on its own, that is, nine subscribers. The three keyword indicators together also give 0.0357, and the seal indicator alone also gives 0.0357 — the other two keywords add nothing. The source of the number is clear: the seal trace is recorded in 0.453 of the suspicious and 0.096 of the others, so this indicator is related to the label. The vehicle record, on the other hand, was generated independently of the label and its contribution came out exactly zero; the trap indicator behaved as a trap.

The harshest rows are the form measures of the text. The character count loses 0.0357 and the word count loses 0.0516; the two together lose 0.0516 and raise the training number to 0.8942. These two columns are in fact a blurred copy of the keyword indicators: if the note gets longer, there is an extra record in it, but which record is not known. The model uses the length to predict the seal record indirectly on the training set, and on the held-out set this indirect path does not hold. The easy way of extracting a number from a text and the way that works are not the same way.

The number of values extracted from a field does not measure the contribution of that field. Ten numbers were extracted, the entire contribution came from one of them, and the other nine took 0.0159 off in total. Increasing the number of columns per field does not give the model more information, it gives it a larger search space.

The Time of the Note

The seal indicator has statistically the same form as the audit column in the Data Leakage lesson of the Introduction to Machine Learning course: both are indicator columns that appear more often in suspicious subscribers. The only difference between them is time. The audit assignment was written after the label became final, and that is why it was leakage; the field note is written during the reading period, before the label becomes final, and is on record at the moment the prediction is to be given.

This distinction cannot be made by looking at the table. Inflation and contribution carry the same sign on the held-out set and appear in the same column. Only the writing order of the source system gives the distinction: after which event is this value recorded. The question itself was built in the Introduction to Machine Learning course; this lesson makes it part of a feature generation decision. The question is asked again for every indicator extracted from text, because the same text field can carry both sentences written during the period and sentences written after it.

Methods that turn a text entirely into numbers, such as the bag of words and the embedding space, are not the subject of this course; they are covered in the Natural Language Processing and Computer Vision course. What is measured here is a narrower and more common decision: extracting a few indicators from a free-text field when one is at hand. The measurement shows that this decision, too, can come out without contribution.

Summary

• Five features were extracted from a date and five from a short note; the date block lost 0.0198, the text block gave a contribution of 0.0317, and the ten numbers together gave 0.0198.
• The whole contribution comes from a single indicator: the seal trace alone adds 0.0357, the three keyword indicators together give the same number, and the vehicle indicator, generated independently of the label, gives exactly zero.
• The form measures of the text are the columns that lose the most: the character count 0.0357, the word count 0.0516; both are a blurred copy of the keyword indicator and raise the training number to 0.8942.
• A noise column with many values is more dangerous than one with few; the day-of-year column offers 366 threshold candidates, and one that looks good on the training set is easy to find.
• The seal indicator and the audit column of the Introduction to Machine Learning course have the same form; only the time of the record separates them, and this question is asked again for every text feature.

Next Step

In two lessons the column count rose from eleven to twenty-two, and each time the measurement said the same thing: most columns either add nothing or lose. Up to here the columns were eliminated one by one by hand. The elimination itself can be tied to a routine — columns can be filtered by their relation to the label, tried by training the model again and again, or the model’s own feature importance can be read. How many columns the three methods select on the same set, how much the held-out number inflates when the selection is made from the whole set, and what happens when it is made from training only — the next lesson measures this.