--- title: 'Feature Stores' source: 'https://academia.sh/en/courses/mlops-fundamentals/feature-stores' course: 'MLOps Fundamentals' language: en updated: '2026-10-21T05:03:02+00:00' license: 'CC BY-SA 4.0' objectives: - 'Defines that the training path and the serving path can compute the same feature differently and measures the gap as training–serving skew.' - 'Shows that test accuracy drops while the model does not change at all and breaks the drop down into the number of flipped predictions.' - "Compares a feature store's closing of the skew with its own cost and writes the stored values, the added stage, and the number of places the rule lives." --- # Feature Stores Measuring whether a feature is computed the same way between training and inference: with the same weights the test accuracy comes out 0.8067 through the training path and 0.7800 through the serving path, and the model never changes. The whole 0.0267 in between comes from which history the feature is computed with; it flips the prediction in 22 of the 600 test rows, breaking 19 and fixing three, and closes entirely when the feature window is raised to two. A feature store reduces the two computations to one and zeroes the skew; in return it brings the cost of a store of 6600 values, a fourth stage in the model branch, and rewriting everything stored when the feature code changes. In three lessons, what was versioned was **the run itself**: which parameters it ran with, which metric it produced, and which data version it read went into the record. All of these records look at the run **from the outside**, and none of them reached the number that goes into the model, that is, **how the feature is computed**. This lesson closes that gap. The code that trains the model runs in one place and the code that produces predictions runs in another; a feature with the same name is computed in both places, and whether the two computations give the same number is something nobody checks. What is to be measured is how that difference shows up in accuracy, and the number the lesson carries is this: **accuracy drops while the model does not change at all.** - **RP41.** The source, the first three stages of the pipeline, and the threshold-based model are **fictional**; no real learning library, feature store product, or serving framework is called. The seed is **20260218**, and the output does not depend on the run. - **RP42.** A **feature** is a number that goes into the model, **computed** from a raw row. There are three here: the subscriber's past consumption mean, the region's past consumption mean, and the commercial indicator. The feature generation procedure was measured in the Data Preparation and Feature Engineering course and is **not repeated**. - **RP43.** The **training path** computes the feature from the **full history**; the **serving path** sees only the **last period**. The difference between the accuracies the two paths give with the same weights is **training–serving skew**. - **RP44.** The label is imbalanced: `{0: 1752, 1: 448}`, the **base class ratio is 0.7964**, and in the test set it is **0.8067**. Accuracy read alone misleads, and in this lesson it will mislead twice. - **RP45.** The model is a threshold with three weights; training is a search on a **27-candidate grid**. Grid search was measured in the Supervised Learning course and is **not repeated**. - **RP46.** **Inference** is producing a prediction for a single row with the trained weights; in this course both training and inference are handled on a **single machine**. - **RP47.** The baseline is the **single-path setup**: the feature is computed separately in two different places, and there is no record saying that the two are the same. - **RP48.** Resolution: the test set is **600 rows** and the smallest step of accuracy is **1/600 = 0.0017**. This lesson's skew is **0.0267**, that is, **sixteen times** the step, and it has been measured. ## Two Computations of the Number That Goes Into the Model A feature is not the raw row itself; it is produced from the raw row by a **rule**. "The subscriber's past consumption mean" is such a rule, and written as a single sentence it looks complete. What is missing is what the sentence silently assumes: **where the past starts.** In training the answer comes by itself; all twelve periods are on the table and the mean is taken from the full history. In serving the situation is different: the prediction is produced as soon as the period arrives, **only that period** is often at hand, and the feature with the same name is computed with the value of the last period. Both rules are written correctly, both carry the same name, and both produce different numbers for the same row. The block below builds the source, the first three stages of the pipeline that face the model, the label, and the fictional model. The `window` parameter of the `features` function separates the two paths: `None` means the full history and `1` means only the last period. The features are independent of the weights and are computed **once**. ```python # FICTIONAL. Meter readings, the first three stages of the pipeline, and the threshold-based # model are defined within the lesson; no real learning library, feature store product, # or serving framework is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): """Consumption from the period difference. Without a previous period the row cannot be computed.""" table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue try: f = int(k["index"]) - int(prev["index"]) except (TypeError, ValueError): dropped += 1 continue out.append(dict(k, m3=f)) return out, dropped TRAIN = PERIODS[1:9] # 2026-02 .. 2026-09 TEST = PERIODS[9:] # 2026-10 .. 2026-12 THRESHOLD = 40 # consumption above this many m3 counts as "high" def rows(): v, _ = s1_screen(generate_source()) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def label(k): return 1 if k["m3"] > THRESHOLD else 0 def features(records, window=None, digits=4): """window=None -> full history (training path); window=1 -> last period (serving path).""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 feats = {} for k in records: i = PERIODS.index(k["period"]) feats[(k["subscriber"], k["period"])] = { "subscriber_mean": mean(sub[k["subscriber"]], i), "region_mean": mean(reg[k["region"]], i), "commercial": 1 if k["type"] == "commercial" else 0} return feats GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def train(records, feats, grid=None, threshold=THRESHOLD, periods=None): """Training = searching the weight triple on the grid. The feature comes from outside.""" grid = GRID if grid is None else grid periods = TRAIN if periods is None else periods subset = [k for k in records if k["period"] in periods] best, best_w = -1.0, None for w in grid: d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], w) > threshold else 0) == label(k)) o = d / len(subset) if o > best: best, best_w = o, w return {"weights": best_w, "train_accuracy": round(best, 4), "candidates": len(grid), "train_rows": len(subset)} def evaluate(records, model, feats, threshold=THRESHOLD, periods=None): periods = TEST if periods is None else periods subset = [k for k in records if k["period"] in periods] d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > threshold else 0) == label(k)) return round(d / len(subset), 4) ``` ## The Same Weights, Two Accuracies The second block trains the model **once** and tests the resulting weight triple on two paths: the same three numbers, the same threshold, the same test periods. The only thing that changes is which history the feature is computed with. ```python import collections S = rows() TEST_ROWS = [k for k in S if k["period"] in TEST] KEYS = [(k["subscriber"], k["period"]) for k in TEST_ROWS] TRUTH = {(k["subscriber"], k["period"]): label(k) for k in TEST_ROWS} FEATS_TRAIN = features(S) # full history -> training path FEATS_SERVE = features(S, window=1) # last period -> serving path MODEL = train(S, FEATS_TRAIN) dist = collections.Counter(label(k) for k in S) base = collections.Counter(TRUTH.values()) print("rows:", len(S), " train:", MODEL["train_rows"], " test:", len(TEST_ROWS)) print("label distribution:", dict(sorted(dist.items())), " base class ratio:", round(dist[0] / len(S), 4), " in test:", round(base[0] / len(TEST_ROWS), 4)) print("model weights:", MODEL["weights"], " candidates:", MODEL["candidates"], " train accuracy:", MODEL["train_accuracy"]) def predictions(feats): return {a: (1 if score(feats[a], MODEL["weights"]) > THRESHOLD else 0) for a in KEYS} PT, PS = predictions(FEATS_TRAIN), predictions(FEATS_SERVE) print(f"\n{'path':<10}{'features differ':>17}{'said high':>11}{'true positive':>15}{'accuracy':>10}") for name, feats, p in (("training", FEATS_TRAIN, PT), ("serving", FEATS_SERVE, PS)): print(f"{name:<10}{sum(1 for a in KEYS if feats[a] != FEATS_TRAIN[a]):>17}" f"{sum(p.values()):>11}{sum(1 for a in KEYS if p[a] == 1 == TRUTH[a]):>15}" f"{evaluate(S, MODEL, feats):>10.4f}") broken = sum(1 for a in KEYS if PT[a] == TRUTH[a] != PS[a]) fixed = sum(1 for a in KEYS if PS[a] == TRUTH[a] != PT[a]) print("changed predictions:", broken + fixed, " broken:", broken, " fixed:", fixed, " net:", broken - fixed) print("skew:", round(evaluate(S, MODEL, FEATS_TRAIN) - evaluate(S, MODEL, FEATS_SERVE), 4), "= net / 600 ; accuracy step 1/600 =", round(1 / len(TEST_ROWS), 4)) ``` ``` rows: 2200 train: 1600 test: 600 label distribution: {0: 1752, 1: 448} base class ratio: 0.7964 in test: 0.8067 model weights: (0.6, 0.4, 0) candidates: 27 train accuracy: 0.7931 path features differ said high true positive accuracy training 0 0 0 0.8067 serving 600 22 3 0.7800 changed predictions: 22 broken: 19 fixed: 3 net: 16 skew: 0.0267 = net / 600 ; accuracy step 1/600 = 0.0017 ``` The weight triple `(0.6, 0.4, 0)` is the same in both rows. Accuracy drops **from 0.8067 to 0.7800**, and the whole drop comes from the feature path; not a single number in the model has changed. The breakdown shows how the difference arises: **the feature is different in 600 of the 600 rows**, but the prediction flips in only **22**, because a difference changes the prediction only when it can carry it to the other side of the threshold. Of the 22 flipped predictions **19 break**, **three get fixed**, a net of **16 rows** is lost, and 16/600 comes to exactly **0.0267**: sixteen times the resolution, that is, a measured quantity. This is also where accuracy misleads for the second time. The base class ratio of the test set is **0.8067**; the `said high` column is **zero** on the training path. That is, the model scoring 0.8067 says "high" to no row at all and produces exactly the number of a model that **says the majority class**. Neither number shows that a model is useful; what is measured is **how far the two paths diverge**. ## Where the Skew Comes From The difference is not a bug; as the history the computation sees gets shorter, the mean approaches the volatility of a single period. The third block sweeps the window from 1 to the full history and tests with the same weights at every step. Next to it is placed a model **trained** on the serving path. ```python FULL = evaluate(S, MODEL, FEATS_TRAIN) print(f"{'window':<9}{'features differ':>17}{'accuracy':>10}{'skew':>9}") for p in (1, 2, 3, 6, None): featsp = features(S, window=p) print(f"{('full' if p is None else p):<9}" f"{sum(1 for a in KEYS if featsp[a] != FEATS_TRAIN[a]):>17}" f"{evaluate(S, MODEL, featsp):>10.4f}{FULL - evaluate(S, MODEL, featsp):>9.4f}") SERVE_MODEL = train(S, FEATS_SERVE) print("\nmodel trained on the serving path:", SERVE_MODEL["weights"], " train:", SERVE_MODEL["train_accuracy"], " test via serving path:", f"{evaluate(S, SERVE_MODEL, FEATS_SERVE):.4f}") ``` ``` window features differ accuracy skew 1 600 0.7800 0.0267 2 600 0.8067 0.0000 3 600 0.8067 0.0000 6 600 0.8067 0.0000 full 0 0.8067 0.0000 model trained on the serving path: (0.6, 0.0, 0) train: 0.7925 test via serving path: 0.8067 ``` The skew **closes entirely at window 2** and does not open again at wider windows. This says the problem is not "the shortness of the history" but its **closeness to zero**. The second column sharpens it: at windows 2, 3, and 6 the feature is **still different in 600 of the 600 rows**, yet the accuracy is exactly the same. The size of the skew is measured not by the difference in feature values but by **how many rows that difference carries to the other side of the threshold**. The last line shows a second repair. When the model is trained with the short-window serving path itself, it finds a different weight `(0.6, 0.0, 0)` and, tested through the same short path, again gives **0.8067**. The moment the inconsistency is lifted, the number comes back; what decides is not which path is chosen but **training and serving using the same path**. ## The Feature Store and Its Cost A **feature store** closes this inconsistency with a **structure** instead of a rule: the feature is computed once, written under the `(subscriber, period)` key, and both training and inference **read** from the same key. Nobody has to check that the two computations agree, because there are not two computations. The fourth block writes the store's own items next to the closed skew. ```python # Feature store: the value is computed ONCE , written under the (subscriber , period) key , # and both training and serving read from the SAME key. STORE = features(S) FIELDS = ("subscriber_mean", "region_mean", "commercial") def from_store(key): return STORE[key] last = TEST[-1] print("store keys:", len(STORE), " fields:", len(FIELDS), " stored values:", len(STORE) * len(FIELDS)) print("read by serving in one period:", sum(1 for a in STORE if a[1] == last), "keys ,", sum(1 for a in STORE if a[1] == last) * len(FIELDS), "values") print("stages of the model branch:", 3, "->", 4, " places the feature rule lives:", 2, "->", 1) print("values rewritten when the feature code changes:", len(STORE) * len(FIELDS)) for name, feats in (("two paths , two computations", FEATS_SERVE), ("one store , one computation", STORE)): print(f"{name:<28} serving accuracy: {evaluate(S, MODEL, feats):.4f}" f" skew: {FULL - evaluate(S, MODEL, feats):.4f}") print("value read from the store the same as in training:", all(from_store(a) == FEATS_TRAIN[a] for a in KEYS)) ``` ``` store keys: 2200 fields: 3 stored values: 6600 read by serving in one period: 200 keys , 600 values stages of the model branch: 3 -> 4 places the feature rule lives: 2 -> 1 values rewritten when the feature code changes: 6600 two paths , two computations serving accuracy: 0.7800 skew: 0.0267 one store , one computation serving accuracy: 0.8067 skew: 0.0000 value read from the store the same as in training: True ``` The gain is a single line: the skew falls **from 0.0267 to 0.0000**, and every value read from the store is exactly the same as the one in training. The cost has four items. The model-facing branch of the pipeline goes **from three stages to four**, because writing the feature is a separate step. **6600 values** are stored, while one period of inference reads only **600** of them; the rest is there so that training can see the history. When the feature rule changes, **all 6600 values** have to be rewritten, otherwise the store carries the old rule and the new rule side by side. The fourth cannot be counted but is real: serving now depends on the store having been written. This trade is the counterpart of the course's rule. Without the store, 0.8067 was truly a number that had been produced, but because it was not written down **by which path** it was produced, it did not come back in serving. A number that cannot be reproduced is not a measurement but a memory. ## Summary - A feature is not a raw row but a number produced from a raw row by a rule; the rule silently answers the question "where does the past start", and training and serving can give it different answers. - With the same weight triple the accuracy is **0.8067** through the training path and **0.7800** through the serving path; the model has not changed at all, and the whole of the **0.0267** training–serving skew comes from the feature path. - Although the skew changes the feature in 600 of 600 rows, it flips the prediction in only **22** rows: 19 breaks, three fixes, a net of 16 rows, that is, 16/600. - The label is imbalanced (`{0: 1752, 1: 448}`, base ratio 0.7964, 0.8067 in the test) and the model scoring 0.8067 says "high" to no row; accuracy alone hides this. - The skew closes entirely at window 2; a feature store gives the same result with a single computation, and its cost is 6600 stored values, a stage that rises to four, and rewriting all of it when the rule changes. ## Next Step The feature store pinned one item: **by which code** the feature is computed now sits in a single place. But that was the only thing pinned. There is no record saying that the data slice read, the test split, the grid searched, and the number of digits the numbers are rounded to have not changed when the same job is run again, and a change in any one of them can change the result. The next lesson counts these five items one by one: it measures how many of them, when pinned, make three runs **exactly** the same, and what kind of assurance a pinning whose contribution cannot be measured gives.