Feature Stores

In three lessons, what was versioned was the run itself: which parameters it ran with, which metric it produced, and which data version it read went into the record. All of these records look at the run from the outside, and none of them reached the number that goes into the model, that is, how the feature is computed.

This lesson closes that gap. The code that trains the model runs in one place and the code that produces predictions runs in another; a feature with the same name is computed in both places, and whether the two computations give the same number is something nobody checks. What is to be measured is how that difference shows up in accuracy, and the number the lesson carries is this: accuracy drops while the model does not change at all.

• RP41. The source, the first three stages of the pipeline, and the threshold-based model are fictional; no real learning library, feature store product, or serving framework is called. The seed is 20260218, and the output does not depend on the run.
• RP42. A feature is a number that goes into the model, computed from a raw row. There are three here: the subscriber’s past consumption mean, the region’s past consumption mean, and the commercial indicator. The feature generation procedure was measured in the Data Preparation and Feature Engineering course and is not repeated.
• RP43. The training path computes the feature from the full history; the serving path sees only the last period. The difference between the accuracies the two paths give with the same weights is training–serving skew.
• RP44. The label is imbalanced: {0: 1752, 1: 448}, the base class ratio is 0.7964, and in the test set it is 0.8067. Accuracy read alone misleads, and in this lesson it will mislead twice.
• RP45. The model is a threshold with three weights; training is a search on a 27-candidate grid. Grid search was measured in the Supervised Learning course and is not repeated.
• RP46. Inference is producing a prediction for a single row with the trained weights; in this course both training and inference are handled on a single machine.
• RP47. The baseline is the single-path setup: the feature is computed separately in two different places, and there is no record saying that the two are the same.
• RP48. Resolution: the test set is 600 rows and the smallest step of accuracy is 1/600 = 0.0017. This lesson’s skew is 0.0267, that is, sixteen times the step, and it has been measured.

Two Computations of the Number That Goes Into the Model

A feature is not the raw row itself; it is produced from the raw row by a rule. “The subscriber’s past consumption mean” is such a rule, and written as a single sentence it looks complete. What is missing is what the sentence silently assumes: where the past starts.

In training the answer comes by itself; all twelve periods are on the table and the mean is taken from the full history. In serving the situation is different: the prediction is produced as soon as the period arrives, only that period is often at hand, and the feature with the same name is computed with the value of the last period. Both rules are written correctly, both carry the same name, and both produce different numbers for the same row.

The block below builds the source, the first three stages of the pipeline that face the model, the label, and the fictional model. The window parameter of the features function separates the two paths: None means the full history and 1 means only the last period. The features are independent of the weights and are computed once.

# FICTIONAL. Meter readings, the first three stages of the pipeline, and the threshold-based
# model are defined within the lesson; no real learning library, feature store product,
# or serving framework is called.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


def s1_screen(records):
    REQUIRED = ("subscriber", "period", "region", "index", "unit")
    keep = [k for k in records if all(a in k for a in REQUIRED)]
    return keep, len(records) - len(keep)


def s2_dedupe(records):
    seen, keep = set(), []
    for k in records:
        key = (k["subscriber"], k["period"])
        if key in seen:
            continue
        seen.add(key)
        keep.append(k)
    return keep, len(records) - len(keep)


def s3_consumption(records):
    """Consumption from the period difference. Without a previous period the row cannot be computed."""
    table = {(k["subscriber"], k["period"]): k for k in records}
    out, dropped = [], 0
    for k in records:
        i = PERIODS.index(k["period"])
        if i == 0:
            continue
        prev = table.get((k["subscriber"], PERIODS[i - 1]))
        if prev is None:
            dropped += 1
            continue
        try:
            f = int(k["index"]) - int(prev["index"])
        except (TypeError, ValueError):
            dropped += 1
            continue
        out.append(dict(k, m3=f))
    return out, dropped


TRAIN = PERIODS[1:9]        # 2026-02 .. 2026-09
TEST = PERIODS[9:]          # 2026-10 .. 2026-12
THRESHOLD = 40              # consumption above this many m3 counts as "high"


def rows():
    v, _ = s1_screen(generate_source())
    v, _ = s2_dedupe(v)
    v, _ = s3_consumption(v)
    return v


def label(k):
    return 1 if k["m3"] > THRESHOLD else 0


def features(records, window=None, digits=4):
    """window=None -> full history (training path); window=1 -> last period (serving path)."""
    ordered = sorted(records, key=lambda k: PERIODS.index(k["period"]))
    sub, reg = {}, {}
    for k in ordered:
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def mean(series, i):
        chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))]
        return round(sum(chosen) / len(chosen), digits) if chosen else 0.0

    feats = {}
    for k in records:
        i = PERIODS.index(k["period"])
        feats[(k["subscriber"], k["period"])] = {
            "subscriber_mean": mean(sub[k["subscriber"]], i),
            "region_mean": mean(reg[k["region"]], i),
            "commercial": 1 if k["type"] == "commercial" else 0}
    return feats


GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]


def score(o, w):
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


def train(records, feats, grid=None, threshold=THRESHOLD, periods=None):
    """Training = searching the weight triple on the grid. The feature comes from outside."""
    grid = GRID if grid is None else grid
    periods = TRAIN if periods is None else periods
    subset = [k for k in records if k["period"] in periods]
    best, best_w = -1.0, None
    for w in grid:
        d = sum(1 for k in subset
                if (1 if score(feats[(k["subscriber"], k["period"])], w) > threshold else 0) == label(k))
        o = d / len(subset)
        if o > best:
            best, best_w = o, w
    return {"weights": best_w, "train_accuracy": round(best, 4),
            "candidates": len(grid), "train_rows": len(subset)}


def evaluate(records, model, feats, threshold=THRESHOLD, periods=None):
    periods = TEST if periods is None else periods
    subset = [k for k in records if k["period"] in periods]
    d = sum(1 for k in subset
            if (1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > threshold else 0)
            == label(k))
    return round(d / len(subset), 4)

The Same Weights, Two Accuracies

The second block trains the model once and tests the resulting weight triple on two paths: the same three numbers, the same threshold, the same test periods. The only thing that changes is which history the feature is computed with.

import collections

S = rows()
TEST_ROWS = [k for k in S if k["period"] in TEST]
KEYS = [(k["subscriber"], k["period"]) for k in TEST_ROWS]
TRUTH = {(k["subscriber"], k["period"]): label(k) for k in TEST_ROWS}
FEATS_TRAIN = features(S)                      # full history  -> training path
FEATS_SERVE = features(S, window=1)            # last period   -> serving path
MODEL = train(S, FEATS_TRAIN)

dist = collections.Counter(label(k) for k in S)
base = collections.Counter(TRUTH.values())
print("rows:", len(S), " train:", MODEL["train_rows"], " test:", len(TEST_ROWS))
print("label distribution:", dict(sorted(dist.items())),
      " base class ratio:", round(dist[0] / len(S), 4),
      " in test:", round(base[0] / len(TEST_ROWS), 4))
print("model weights:", MODEL["weights"], " candidates:", MODEL["candidates"],
      " train accuracy:", MODEL["train_accuracy"])


def predictions(feats):
    return {a: (1 if score(feats[a], MODEL["weights"]) > THRESHOLD else 0) for a in KEYS}


PT, PS = predictions(FEATS_TRAIN), predictions(FEATS_SERVE)
print(f"\n{'path':<10}{'features differ':>17}{'said high':>11}{'true positive':>15}{'accuracy':>10}")
for name, feats, p in (("training", FEATS_TRAIN, PT), ("serving", FEATS_SERVE, PS)):
    print(f"{name:<10}{sum(1 for a in KEYS if feats[a] != FEATS_TRAIN[a]):>17}"
          f"{sum(p.values()):>11}{sum(1 for a in KEYS if p[a] == 1 == TRUTH[a]):>15}"
          f"{evaluate(S, MODEL, feats):>10.4f}")
broken = sum(1 for a in KEYS if PT[a] == TRUTH[a] != PS[a])
fixed = sum(1 for a in KEYS if PS[a] == TRUTH[a] != PT[a])
print("changed predictions:", broken + fixed, " broken:", broken, " fixed:", fixed,
      " net:", broken - fixed)
print("skew:", round(evaluate(S, MODEL, FEATS_TRAIN) - evaluate(S, MODEL, FEATS_SERVE), 4),
      "= net / 600 ; accuracy step 1/600 =", round(1 / len(TEST_ROWS), 4))

rows: 2200  train: 1600  test: 600
label distribution: {0: 1752, 1: 448}  base class ratio: 0.7964  in test: 0.8067
model weights: (0.6, 0.4, 0)  candidates: 27  train accuracy: 0.7931

path        features differ  said high  true positive  accuracy
training                  0          0              0    0.8067
serving                 600         22              3    0.7800
changed predictions: 22  broken: 19  fixed: 3  net: 16
skew: 0.0267 = net / 600 ; accuracy step 1/600 = 0.0017

The weight triple (0.6, 0.4, 0) is the same in both rows. Accuracy drops from 0.8067 to 0.7800, and the whole drop comes from the feature path; not a single number in the model has changed. The breakdown shows how the difference arises: the feature is different in 600 of the 600 rows, but the prediction flips in only 22, because a difference changes the prediction only when it can carry it to the other side of the threshold. Of the 22 flipped predictions 19 break, three get fixed, a net of 16 rows is lost, and 16/600 comes to exactly 0.0267: sixteen times the resolution, that is, a measured quantity.

This is also where accuracy misleads for the second time. The base class ratio of the test set is 0.8067; the said high column is zero on the training path. That is, the model scoring 0.8067 says “high” to no row at all and produces exactly the number of a model that says the majority class. Neither number shows that a model is useful; what is measured is how far the two paths diverge.

Where the Skew Comes From

The difference is not a bug; as the history the computation sees gets shorter, the mean approaches the volatility of a single period. The third block sweeps the window from 1 to the full history and tests with the same weights at every step. Next to it is placed a model trained on the serving path.

FULL = evaluate(S, MODEL, FEATS_TRAIN)
print(f"{'window':<9}{'features differ':>17}{'accuracy':>10}{'skew':>9}")
for p in (1, 2, 3, 6, None):
    featsp = features(S, window=p)
    print(f"{('full' if p is None else p):<9}"
          f"{sum(1 for a in KEYS if featsp[a] != FEATS_TRAIN[a]):>17}"
          f"{evaluate(S, MODEL, featsp):>10.4f}{FULL - evaluate(S, MODEL, featsp):>9.4f}")

SERVE_MODEL = train(S, FEATS_SERVE)
print("\nmodel trained on the serving path:", SERVE_MODEL["weights"],
      " train:", SERVE_MODEL["train_accuracy"],
      " test via serving path:", f"{evaluate(S, SERVE_MODEL, FEATS_SERVE):.4f}")

window     features differ  accuracy     skew
1                      600    0.7800   0.0267
2                      600    0.8067   0.0000
3                      600    0.8067   0.0000
6                      600    0.8067   0.0000
full                     0    0.8067   0.0000

model trained on the serving path: (0.6, 0.0, 0)  train: 0.7925  test via serving path: 0.8067

The skew closes entirely at window 2 and does not open again at wider windows. This says the problem is not “the shortness of the history” but its closeness to zero. The second column sharpens it: at windows 2, 3, and 6 the feature is still different in 600 of the 600 rows, yet the accuracy is exactly the same. The size of the skew is measured not by the difference in feature values but by how many rows that difference carries to the other side of the threshold.

The last line shows a second repair. When the model is trained with the short-window serving path itself, it finds a different weight (0.6, 0.0, 0) and, tested through the same short path, again gives 0.8067. The moment the inconsistency is lifted, the number comes back; what decides is not which path is chosen but training and serving using the same path.

The Feature Store and Its Cost

A feature store closes this inconsistency with a structure instead of a rule: the feature is computed once, written under the (subscriber, period) key, and both training and inference read from the same key. Nobody has to check that the two computations agree, because there are not two computations. The fourth block writes the store’s own items next to the closed skew.

# Feature store: the value is computed ONCE , written under the (subscriber , period) key ,
# and both training and serving read from the SAME key.
STORE = features(S)
FIELDS = ("subscriber_mean", "region_mean", "commercial")


def from_store(key):
    return STORE[key]


last = TEST[-1]
print("store keys:", len(STORE), " fields:", len(FIELDS), " stored values:", len(STORE) * len(FIELDS))
print("read by serving in one period:", sum(1 for a in STORE if a[1] == last), "keys ,",
      sum(1 for a in STORE if a[1] == last) * len(FIELDS), "values")
print("stages of the model branch:", 3, "->", 4, " places the feature rule lives:", 2, "->", 1)
print("values rewritten when the feature code changes:", len(STORE) * len(FIELDS))

for name, feats in (("two paths , two computations", FEATS_SERVE), ("one store , one computation", STORE)):
    print(f"{name:<28} serving accuracy: {evaluate(S, MODEL, feats):.4f}"
          f"  skew: {FULL - evaluate(S, MODEL, feats):.4f}")
print("value read from the store the same as in training:",
      all(from_store(a) == FEATS_TRAIN[a] for a in KEYS))

store keys: 2200  fields: 3  stored values: 6600
read by serving in one period: 200 keys , 600 values
stages of the model branch: 3 -> 4  places the feature rule lives: 2 -> 1
values rewritten when the feature code changes: 6600
two paths , two computations serving accuracy: 0.7800  skew: 0.0267
one store , one computation  serving accuracy: 0.8067  skew: 0.0000
value read from the store the same as in training: True

The gain is a single line: the skew falls from 0.0267 to 0.0000, and every value read from the store is exactly the same as the one in training. The cost has four items. The model-facing branch of the pipeline goes from three stages to four, because writing the feature is a separate step. 6600 values are stored, while one period of inference reads only 600 of them; the rest is there so that training can see the history. When the feature rule changes, all 6600 values have to be rewritten, otherwise the store carries the old rule and the new rule side by side. The fourth cannot be counted but is real: serving now depends on the store having been written.

This trade is the counterpart of the course’s rule. Without the store, 0.8067 was truly a number that had been produced, but because it was not written down by which path it was produced, it did not come back in serving. A number that cannot be reproduced is not a measurement but a memory.

Summary

• A feature is not a raw row but a number produced from a raw row by a rule; the rule silently answers the question “where does the past start”, and training and serving can give it different answers.
• With the same weight triple the accuracy is 0.8067 through the training path and 0.7800 through the serving path; the model has not changed at all, and the whole of the 0.0267 training–serving skew comes from the feature path.
• Although the skew changes the feature in 600 of 600 rows, it flips the prediction in only 22 rows: 19 breaks, three fixes, a net of 16 rows, that is, 16/600.
• The label is imbalanced ({0: 1752, 1: 448}, base ratio 0.7964, 0.8067 in the test) and the model scoring 0.8067 says “high” to no row; accuracy alone hides this.
• The skew closes entirely at window 2; a feature store gives the same result with a single computation, and its cost is 6600 stored values, a stage that rises to four, and rewriting all of it when the rule changes.

Next Step

The feature store pinned one item: by which code the feature is computed now sits in a single place. But that was the only thing pinned. There is no record saying that the data slice read, the test split, the grid searched, and the number of digits the numbers are rounded to have not changed when the same job is run again, and a change in any one of them can change the result. The next lesson counts these five items one by one: it measures how many of them, when pinned, make three runs exactly the same, and what kind of assurance a pinning whose contribution cannot be measured gives.