Feature Stores

Models are only as good as the features they are trained and served with, and keeping those two consistent is surprisingly hard. This page explains what a feature store does, the offline/online split, how point-in-time joins prevent leakage, and when one is worth adopting.

1. What a Feature Store Is

A feature is an input value a model uses, such as "number of orders in the last 30 days" for a customer. A feature store is a system that manages features end to end: it holds their definitions, computes or ingests their values, stores history for training, serves the latest values at low latency for inference, and records metadata so features can be found and reused.

The diagram shows the standard two-store design. The offline store (usually tables in a warehouse or lake) keeps every historical value with a timestamp; it is optimised for large scans to build training sets. The online store (a key-value database such as Redis, DynamoDB or Cassandra) keeps only the latest value per entity key and answers lookups in milliseconds. A materialisation job copies recent values from offline to online.

2. Why Teams Use One

Three recurring problems motivate feature stores:

• Training-serving skew. When data scientists compute features in a notebook and engineers re-implement them in the serving service, small differences (time zones, null handling, window boundaries) mean the model sees different inputs in production than in training. A single definition used by both paths removes that class of bug.
• Label leakage. Building a training set requires each feature value as it was at the moment of the prediction, not as it is today. Getting this right by hand is error-prone; feature stores make point-in-time joins a built-in operation.
• Duplication. Without a catalogue, every team rebuilds "customer lifetime value" slightly differently. A registry with owners and documentation encourages reuse.

A feature store is not always needed. A single batch model scored nightly can be served from a warehouse table. The value appears with real-time inference, many models sharing features, or several teams.

3. How Point-in-Time Joins Work

Training data is a table of label events: entity key, timestamp, and outcome (for example, "customer 1, 9 September, did not churn"). For each row, the feature store finds, per feature, the most recent value whose timestamp is at or before the event timestamp. Formally, for label time t it selects the value with max(feature_ts) where feature_ts ≤ t, optionally ignoring values older than a time-to-live (TTL).

This is an "as-of" join. In SQL it can be written with a window function or a lateral join; dataframe libraries offer it directly (merge_asof in pandas, join_asof in Polars), and Spark-based stores implement it at scale. The diagram shows why it matters: a naive equality join on customer_id would attach the day-12 value to a day-9 label, leaking information the model will not have when predicting.

A second timing subtlety is availability time. If the day-5 value was computed by a batch job that only finished on day 6, it was not truly available on day 5. Careful stores track both the event timestamp and when the value was written.

Batch, streaming and on-demand features

Features differ in how they are computed, and a mature store supports all three kinds. Batch features are computed on a schedule from the warehouse (for example, lifetime spend recomputed nightly); they are cheap but can be up to a day old. Streaming features are updated from an event stream within seconds (for example, the number of failed logins in the last ten minutes) and are written to both stores as they change. On-demand features are computed at request time from inputs only available then, such as the distance between the user's current location and a merchant. The store's job in every case is the same: one definition, versioned in code, producing identical values whether it is called while building a training set or while answering a live request.

Freshness, cost and complexity rise in that order, so default to batch features and promote a feature to streaming only when an experiment shows the fresher value actually improves the model.

4. Worked Example

The heart of an offline store in a few lines of pandas:

# Point-in-time join with pandas.merge_asof: the core operation of an offline store.
import pandas as pd

features = pd.DataFrame({
    "customer_id": [1, 1, 1, 2, 2],
    "feature_ts":  pd.to_datetime(["2026-09-01", "2026-09-05", "2026-09-12",
                                   "2026-09-02", "2026-09-08"]),
    "orders_30d":  [2, 4, 9, 1, 3],
}).sort_values("feature_ts")

labels = pd.DataFrame({
    "customer_id": [1, 2],
    "event_ts":    pd.to_datetime(["2026-09-09", "2026-09-09"]),
    "churned":     [0, 1],
}).sort_values("event_ts")

train = pd.merge_asof(labels, features,
                      left_on="event_ts", right_on="feature_ts",
                      by="customer_id", direction="backward",       # only past values
                      tolerance=pd.Timedelta("30D"))                # ignore stale values
print(train[["customer_id", "event_ts", "orders_30d", "churned"]])
#    customer_id   event_ts  orders_30d  churned
# 0            1 2026-09-09           4        0
# 1            2 2026-09-09           3        1

Customer 1's label on 9 September picks up orders_30d = 4 (the 5 September value), not 9 from 12 September. The same idea expressed with Feast, an open-source feature store, declares the entity, source and feature view once, then uses one call for training data and another for online lookups:

# Feast feature definition (feature_repo/features.py) - API as of Feast 0.3x+
from datetime import timedelta
from feast import Entity, FeatureView, Field, FileSource
from feast.types import Int64

customer = Entity(name="customer", join_keys=["customer_id"])
source = FileSource(path="data/customer_stats.parquet", timestamp_field="feature_ts")

customer_stats = FeatureView(
    name="customer_stats", entities=[customer], ttl=timedelta(days=30),
    schema=[Field(name="orders_30d", dtype=Int64)], source=source)

# Training:  store.get_historical_features(entity_df=labels, features=["customer_stats:orders_30d"])
# Serving:   store.get_online_features(features=[...], entity_rows=[{"customer_id": 1}])

Feast's API changes between releases, so check the documentation for your version; the concepts (entity, feature view, source, TTL, historical vs online retrieval) are stable across tools.

5. Comparison of Options

Choose by where your data and models already live; the integration cost usually outweighs feature differences.

6. Pitfalls

• Building one too early. A feature store is infrastructure to operate. Start with well-named warehouse tables and adopt one when online serving or reuse demands it.
• Ignoring freshness. Online values are only as fresh as the last materialisation. Monitor lag and define SLAs per feature.
• On-demand features outside the store. Features computed at request time (such as "seconds since last click") still need one shared implementation, or skew returns.
• Wrong TTL. Too short and training rows get nulls; too long and stale values look current.
• Entity key mismatch. Different ID formats between training and serving produce silent misses that return default values.
• No backfill story. A new feature is useless for training until history is computed. Plan backfills as part of adding a feature.

7. Related Topics

• Experiment Tracking — recording which features and data produced each model
• Idempotent Pipelines and Backfills — backfilling feature history safely
• Batch vs Streaming — batch versus streaming feature computation
• Bias-Variance Trade-off — why leakage inflates offline metrics
• Real-Time Embedding Pipeline — a streaming feature pipeline in practice
• Machine Learning — section home

8. Further Reading

• Hermann and Del Balso (2017), Meet Michelangelo: Uber's Machine Learning Platform (Uber Engineering blog), an early public description of a feature store.
• Sculley et al. (2015), Hidden Technical Debt in Machine Learning Systems, NeurIPS.
• Chip Huyen, Designing Machine Learning Systems (O'Reilly, 2022), chapters on feature engineering and data distribution shifts.
• Feast documentation: concepts (entities, feature views, point-in-time joins).
• pandas documentation: merge_asof.