--- title: 'Time and Text Features' source: 'https://academia.sh/en/courses/feature-engineering/time-and-text-features' course: 'Data Preparation and Feature Engineering' language: en updated: '2026-10-07T04:31:19+00:00' license: 'CC BY-SA 4.0' objectives: - 'Generates five features from a date field and five from a short text field and measures the total contribution of the two blocks separately on the held-out set' - 'Shows that there is no link between the number of values generated per field and the contribution: the total contribution of ten numbers stays below the contribution of a single indicator' - 'Separates whether a feature extracted from text is leakage by the time question: is the value at hand at the moment the prediction is to be given' --- # Time and Text Features Extracting five numbers each from a date and from a short text and counting their total contributions on the held-out set: while the line before the decision is 0.7897, the five numbers from the date lose 0.0198, the five numbers from the note add 0.0317, and the ten numbers together add 0.0198. The whole contribution comes from a single indicator; the seal trace indicator alone adds 0.0357 while the text length loses 0.0357 and the word count loses 0.0516. How many numbers a field yields does not determine the contribution; the relation of the field to the label does. The previous lesson generated six features, measured all six one by one, and left a single column behind: consumption per person, the ratio feature that carried the line before the decision from 0.7778 to 0.7897. The other five were removed from the set. From this lesson on, the line before the decision is the line of **twelve columns**. The subscriber table has two more fields that have not been given to the model, and neither is a number: the **date** on which the contract started and the **short note** the field crew left during the reading. A decision tree cannot split either of them as they are; a date is an ordered value, but its ordering does not carry the seasonality of the year, and a text is not ordered at all. Several numbers can be extracted from each of them, and this is exactly the measure of this lesson: **how many numbers a field yields and what the total contribution of those numbers is.** - **FS8.** The line before the decision is twelve columns: the eleven columns of the Introduction to Machine Learning course and `per_person`. The model does not change: a decision tree of depth 6. The baseline is 0.7579. - **FS9.** The contract date and the field note are **fictional**; they are generated from the same seed (20260218) depending on the subscriber id and added to the table in this lesson. - **FS10.** The field note is written **during the reading period**; the suspicion label becomes final after the period closes. The content of the note is at hand at prediction time. - **FS11.** The seal trace record enters the note with probability 0.42 for suspicious subscribers and 0.10 for the others; the access record enters with probability 0.30 if the period count is below three and 0.08 otherwise. - **FS12.** The extra note enters with probability 0.25 and is one of three equally likely topics; the vehicle record thus falls into the note with a probability of about 0.08 and is **independent** of the label; it is a trap indicator. - **FS13.** String operations were built in the data frames topic of the Data Analysis with Python course; the string routine is not repeated here, only the number extracted from the text is measured. The bag of words and the embedding space are not the subject of this course. - **FS14.** Contribution is the difference of the twelve columns' number on the held-out set from the line before the decision; one row is 0.0040. ## Five Numbers from a Date, Five from a Note The setup produces the same set as the previous lessons; the only difference is that the `per_person` column, whose persistence has been measured, is added from the start. ```python # COMMON — MODEL: the FICTIONAL subscriber table of the Introduction to Machine Learning course is rebuilt with the same seed; # the model is a decision tree of depth 6. The only new column left from the previous lesson is per_person. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) BASE = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] BASE_FIELDS = BASE + ["per_person"] for x in DATA: x["per_person"] = round(x["mean_consumption"] / x["household"], 3) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] def gini(rows): p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0 return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): p, best = sum(x["suspicious"] for x in rows) / len(rows), None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = gini(rows) - (len(left) * gini(left) + len(right) * gini(right)) / len(rows) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields, min_size), "right": tree(best[4], depth - 1, fields, min_size)} def predict(node, x): while "prediction" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["prediction"] def accuracy(node, rows): return sum(predict(node, x) == x["suspicious"] for x in rows) / len(rows) def baseline(train, rows): c = int(sum(x["suspicious"] for x in train) * 2 > len(train)) return sum(x["suspicious"] == c for x in rows) / len(rows) def measure(fields, depth=6): m = tree(TRAIN, depth, fields) return accuracy(m, TRAIN), accuracy(m, TEST) ``` The five numbers extracted from the date correspond to two different questions. The month, the quarter, the day of the week and the day of the year are **seasonal** indicators: each is a number that cycles within itself, and their purpose is to catch a seasonal pattern. The month and the quarter give the same value on the same calendar date every year; the day of the week shifts from year to year, and the day of the year moves by one day in a leap year. The contract age, on the other hand, is a **linear** number and grows every day. The five numbers extracted from the note also fall into two groups: two of them measure the **form** of the text (character count, word count), and three show the presence of a particular **keyword**. ```python import datetime START = datetime.date(2024, 1, 1) CORE = ["meter access completed", "reading okay", "subscriber was out", "dirty cover", "readings repeated", "valve was inspected"] EXTRA = ["car parked nearby", "doorbell not working", "garden gates open"] for x in DATA: r = rng(SEED + 61000 + x["id"]) x["contract_day"] = int(r() * 1460) g = START + datetime.timedelta(days=x["contract_day"]) x["month"], x["quarter"] = g.month, (g.month - 1) // 3 + 1 x["weekday"], x["day_of_year"] = g.weekday(), g.timetuple().tm_yday x["contract_age"] = 1460 - x["contract_day"] p = [CORE[int(r() * len(CORE))]] if r() < (0.42 if x["suspicious"] else 0.10): p.append("seal was torn") if r() < (0.30 if x["period"] < 3 else 0.08): p.append("access unavailable") if r() < 0.25: p.append(EXTRA[int(r() * len(EXTRA))]) n = ", ".join(p) x["field_note"], x["note_length"], x["note_words"] = n, len(n), len(n.split()) x["kw_seal"] = int("seal" in n) x["kw_access"] = int("access unavailable" in n) x["kw_car"] = int("car" in n) TIME = ["month", "quarter", "weekday", "day_of_year", "contract_age"] TEXT = ["note_length", "note_words", "kw_seal", "kw_access", "kw_car"] for i in (4, 9, 10): print(f"note example: {DATA[i]['field_note']}") print(f"seal trace recorded in {sum(x['kw_seal'] for x in DATA)} rows; in " f"{sum(x['kw_seal'] for x in DATA if x['suspicious']) / sum(x['suspicious'] for x in DATA):.3f}" f" of the suspicious, in " f"{sum(x['kw_seal'] for x in DATA if not x['suspicious']) / sum(1 - x['suspicious'] for x in DATA):.3f}" f" of the others\n") BEFORE = measure(BASE_FIELDS) print(f"{'block':<16} {'columns':>7} {'train':>9} {'held_out':>9} {'contrib':>8}") print(f"{'-- none --':<16} {0:>7} {BEFORE[0]:>9.4f} {BEFORE[1]:>9.4f}") for name, fs in (("from date", TIME), ("from text", TEXT), ("both", TIME + TEXT)): e, s = measure(BASE_FIELDS + fs) print(f"{name:<16} {len(fs):>7} {e:>9.4f} {s:>9.4f} {s - BEFORE[1]:>+8.4f}") ``` ``` note example: readings repeated note example: subscriber was out, access unavailable, car parked nearby note example: dirty cover, seal was torn seal trace recorded in 219 rows; in 0.453 of the suspicious, in 0.096 of the others block columns train held_out contrib -- none -- 0 0.8519 0.7897 from date 5 0.8611 0.7698 -0.0198 from text 5 0.8942 0.8214 +0.0317 both 10 0.8955 0.8095 +0.0198 ``` Side by side, the three numbers show that there is no link between the number of fields and the contribution. Five numbers were extracted from each of two fields; one earned 0.0317, the other lost 0.0198. When ten numbers are given together, the total contribution falls to 0.0198 — that is, the date field takes back three of the eight subscribers the text field earned. The training number, on the other hand, rises in all three setups: from 0.8519 to 0.8955. That the training number rises as the column count grows is a requirement of the definition, not a measurement. That the date block comes out negative is not surprising, because in this fiction no link has been built between the contract date and the suspicion. The cause of the loss is not the absence of a link but **five columns competing as split candidates**: the day-of-year column has 366 distinct values, and among that many threshold candidates it is easy to find one that looks good on the training set. A noise column with many values is more dangerous than a noise column with few. ## Where the Contribution Comes From When the blocks are opened one by one, it becomes visible which column the total numbers come from. ```python print(f"{'single feature':<16} {'source':<8} {'train':>9} {'held_out':>9} {'contrib':>8}") for f in TIME + TEXT: e, s = measure(BASE_FIELDS + [f]) print(f"{f:<16} {'date' if f in TIME else 'text':<8} {e:>9.4f} {s:>9.4f} " f"{s - BEFORE[1]:>+8.4f}") for name, fs in (("only three keywords", ["kw_seal", "kw_access", "kw_car"]), ("only two counters", ["note_length", "note_words"]), ("only the seal", ["kw_seal"])): e, s = measure(BASE_FIELDS + fs) print(f"\n{name:<20} {e:.4f} {s:.4f} {s - BEFORE[1]:+.4f}", end="") print() ``` ``` single feature source train held_out contrib month date 0.8519 0.7897 +0.0000 quarter date 0.8519 0.7897 +0.0000 weekday date 0.8532 0.7778 -0.0119 day_of_year date 0.8519 0.7897 +0.0000 contract_age date 0.8585 0.7817 -0.0079 note_length text 0.8823 0.7540 -0.0357 note_words text 0.8783 0.7381 -0.0516 kw_seal text 0.8876 0.8254 +0.0357 kw_access text 0.8519 0.7857 -0.0040 kw_car text 0.8519 0.7897 +0.0000 only three keywords 0.8876 0.8254 +0.0357 only two counters 0.8942 0.7381 -0.0516 only the seal 0.8876 0.8254 +0.0357 ``` Nine of the ten features give zero or negative contribution. The whole gain comes from the `kw_seal` column: 0.0357 on its own, that is, nine subscribers. The three keyword indicators together also give 0.0357, and the seal indicator alone also gives 0.0357 — the other two keywords add nothing. The source of the number is clear: the seal trace is recorded in 0.453 of the suspicious and 0.096 of the others, so this indicator is related to the label. The vehicle record, on the other hand, was generated independently of the label and its contribution came out exactly zero; the trap indicator behaved as a trap. The harshest rows are the **form** measures of the text. The character count loses 0.0357 and the word count loses 0.0516; the two together lose 0.0516 and raise the training number to 0.8942. These two columns are in fact a blurred copy of the keyword indicators: if the note gets longer, there is an extra record in it, but which record is not known. The model uses the length to predict the seal record indirectly on the training set, and on the held-out set this indirect path does not hold. The easy way of extracting a number from a text and the way that works are not the same way. The number of values extracted from a field does not measure the contribution of that field. Ten numbers were extracted, the entire contribution came from one of them, and the other nine took 0.0159 off in total. Increasing the number of columns per field does not give the model more information, it gives it a larger **search space**. ## The Time of the Note The seal indicator has statistically the same form as the audit column in the Data Leakage lesson of the Introduction to Machine Learning course: both are indicator columns that appear more often in suspicious subscribers. The only difference between them is **time**. The audit assignment was written after the label became final, and that is why it was leakage; the field note is written during the reading period, before the label becomes final, and is on record at the moment the prediction is to be given. This distinction cannot be made by looking at the table. Inflation and contribution carry the same sign on the held-out set and appear in the same column. Only the **writing order** of the source system gives the distinction: after which event is this value recorded. The question itself was built in the Introduction to Machine Learning course; this lesson makes it part of a feature generation decision. The question is asked again for every indicator extracted from text, because the same text field can carry both sentences written during the period and sentences written after it. Methods that turn a text entirely into numbers, such as the bag of words and the embedding space, are not the subject of this course; they are covered in the Natural Language Processing and Computer Vision course. What is measured here is a narrower and more common decision: extracting **a few indicators** from a free-text field when one is at hand. The measurement shows that this decision, too, can come out without contribution. ## Summary - Five features were extracted from a date and five from a short note; the date block lost 0.0198, the text block gave a contribution of 0.0317, and the ten numbers together gave 0.0198. - The whole contribution comes from a single indicator: the seal trace alone adds 0.0357, the three keyword indicators together give the same number, and the vehicle indicator, generated independently of the label, gives exactly zero. - The form measures of the text are the columns that lose the most: the character count 0.0357, the word count 0.0516; both are a blurred copy of the keyword indicator and raise the training number to 0.8942. - A noise column with many values is more dangerous than one with few; the day-of-year column offers 366 threshold candidates, and one that looks good on the training set is easy to find. - The seal indicator and the audit column of the Introduction to Machine Learning course have the same form; only the time of the record separates them, and this question is asked again for every text feature. ## Next Step In two lessons the column count rose from eleven to twenty-two, and each time the measurement said the same thing: most columns either add nothing or lose. Up to here the columns were eliminated **one by one by hand**. The elimination itself can be tied to a routine — columns can be filtered by their relation to the label, tried by training the model again and again, or the model's own feature importance can be read. How many columns the three methods select on the same set, how much the held-out number inflates when the selection is made from **the whole set**, and what happens when it is made **from training only** — the next lesson measures this.