ML Feature Store

Features for training and serving

Design a feature store: one place where ML features are defined once, computed in batch and streaming, served to live models in milliseconds, and replayed exactly as they were for training.

Overview

A feature is a number or value a model uses to make a prediction, such as 'orders this user placed in the last 30 days'. A feature store is the service that defines each feature once, computes it, stores its history for training, and serves its latest value to live models.

Without one, each team writes the same feature twice: once in a notebook or batch job for training, and again in the serving code. The two versions drift apart, and the model sees different inputs in production than it learned from. Google's Rules of Machine Learning calls this training-serving skew and names the fixes this design is built on: reuse the same code in both paths, and log the features used at serving time.

There are two hard parts. The first is time: a training example from January 12 must see the feature as it was on January 12, not today's value. The second is speed: a ranking request may need features for hundreds of items within a few milliseconds.

Questions to ask first

Who reads online features? — Ranking and fraud models, about 10,000 requests a second at peak
How many keys per request? — 1 user plus 200 candidate items, about 50 features each
How fresh must features be? — Streaming features within seconds; batch features within a day
How far back does training look? — Two years of history
Can features be computed at request time? — Yes, for cheap ones that need the request itself (on-demand)
One team or many? — Many teams, sharing and reusing features

What you will be able to do

Explain training-serving skew and the two defenses: one definition, and logged serving features.
Build a point-in-time correct training set, in Python and in SQL, and say what a TTL does to it.
Size and lay out the online store, and keep it correct when streaming events arrive out of order.
Choose between batch, streaming and on-demand computation for a feature.
Handle backfills, new versions and monitoring without breaking models that are already live.

Key concepts

Training-serving skew
Point-in-time join
Online store
Offline store
Materialization
Streaming features
Entity key
Feature versioning
Freshness
Backfill

Functional requirements

Register a feature with its entity key, type, owner, source and transformation, and version it
Compute features in batch (hourly or daily) and from event streams (seconds)
Serve the latest values for a list of entity keys to online models
Build training datasets with the value each feature had at each label's timestamp
Backfill the history of a new feature or a new version
Report freshness, null rate and distribution changes per feature

Non-functional requirements

Online reads: p99 under 10 ms for one request's batch of keys (a target we chose)
Online availability of 99.95% (a target we chose); a missing feature falls back to a default, never an error to the model
The same definition produces the same value offline and online
Training joins never use a value that was not available at the label's time
Streaming features visible online within about 10 seconds of the event (a target we chose)

Interview questions

What is training-serving skew and how does a feature store prevent it?
Walk me through a point-in-time join.
How do you serve 200 items' features in a few milliseconds?
Streaming events arrive late and duplicated. How do you keep the online store right?
Why keep two stores instead of one database?
How do you change a feature without breaking models that use it?
What do you monitor, and what does each alert catch?