Real-Time Feature Database

Fresh features for live models in milliseconds

Design the online side of a feature store: a database that keeps seconds-fresh feature values computed from event streams and returns a full feature vector to a live model within a few milliseconds.

Overview

A live model, such as a fraud check on a card payment, needs features: numbers like 'payments by this card in the last 10 minutes' or 'average order value for this merchant over 30 days'. A real-time feature database is the online store that holds the newest value of every feature for every entity and hands them to the model while the request waits.

It is the online half of a feature store. The ML Feature Store topic covers the whole platform, including the offline history and point-in-time training joins. This topic goes deep on the part an interviewer probes when they say 'real-time': the latency budget, the key layout, streaming windows, freshness and failure.

The hard part is that the two paths pull in opposite directions. Reads want everything precomputed and one hop away. Writes come from stream jobs that see events late, out of order and sometimes twice. If the online value is computed differently from the training value, the model quietly gets worse; Google's Rules of Machine Learning calls this training-serving skew.

Questions to ask first

Who reads, and how fast? — A fraud and a ranking model, about 7,000 predictions a second at peak
How many entities per prediction? — 1 user, 1 merchant and up to 50 candidate items: 52 keys
What latency is left for features? — 10 ms at p99 out of a 100 ms prediction budget (targets we chose)
How fresh must features be? — Streaming counts within about a minute; daily batch features within two days
What if a value is missing? — The model gets the default it was trained with, plus a missing flag
Is losing a few seconds of updates acceptable? — Yes: features can be recomputed from the stream, so the store is not the system of record

What you will be able to do

Split a prediction's latency budget and show why the feature fetch must be pipelined by shard.
Lay out keys and values so one entity is one read, with versions side by side.
Compute sliding-window features by event time and say what a watermark does with late events.
Serve a fresh value, or a default with a flag, and never an error.
Log served features so the next model trains on exactly what production saw.

Key concepts

Online store
Feature vector
Pipelining
Sliding window
Event time
Watermark
Freshness SLA
Default fallback
Served-feature log
Hot key

Functional requirements

Return feature vectors for up to about 50 entities of several types (user, merchant, item) in one call
Accept writes from stream jobs and bulk loads from batch jobs, keeping the newest value by event time
Compute windowed aggregates (counts and sums over the last minutes or days) from event streams
Store a freshness target per feature and report the age of every value served
Serve two versions of a feature view at the same time while a new model rolls out
Log every served vector with the request id and model version

Non-functional requirements

Feature fetch p99 under 10 ms for a 52-entity request (a target we chose)
Online reads available 99.95% of the time (a target we chose); a missing value degrades to a default, never an error
Streaming features visible within about 60 seconds of the event (a target we chose)
Replaying a stream after a failure does not move values backward
Scale reads and writes by adding shards, without a global lock

Interview questions

How do you fetch features for 52 entities within 10 ms?
Why event time and not processing time for streaming features?
What does a watermark do, and what happens to late events?
Streams replay after a crash. How do you keep the online store correct?
What do you return when a feature is missing or stale?
How do you roll out a new version of a feature?
How does training avoid skew with the online store?
Redis or DynamoDB for the online store?