Time Series Forecasting System

Demand Forecasting at Scale

Design a system that forecasts daily demand for a million product-and-store series every night, with a range of likely values for each day, and publishes the results before stores place their orders.

Overview

A retailer orders stock every day for every product in every store. To order well it needs a forecast: how many units of this product will this store sell on each of the next few weeks' days. Multiply 20,000 products by 50 stores and there are a million series (one sequence of daily numbers each), all needing a fresh forecast every night.

Three things make this harder than a normal prediction service. The data is ordered in time, so tomorrow depends on last week, on the season and on holidays, and a model tested on randomly chosen days will look better than it is. The output must be a range: an order for 105 units is a different decision if sales could be 90 to 120 than if they could be 40 to 200. And the whole run is a batch with a deadline: if it is late or half-written, stores order from bad numbers.

Questions to ask first

How many series, at what grain? — 1 million product-and-store series, daily
How far ahead, and how often? — 28 days ahead, refreshed every night
A single number or a range? — Quantiles p10, p50, p90 for every day
Who reads the forecasts, and when? — The ordering system at 6:00 every morning, plus about 2,000 planners during the day
What else drives demand? — Holidays and planned promotions, known in advance
How much history exists? — About 3 years for most series; new products have none

What you will be able to do

Size a nightly forecasting run and say how many workers it needs.
Choose between one model per series and one global model, with reasons.
Produce and score ranges (quantiles) instead of single numbers, and check that an 80% band really holds 80% of outcomes.
Backtest without letting the future leak into the past.
Publish a nightly run all or nothing, and handle drift, late data and brand-new products.

Key concepts

Local vs global models
Seasonality
Quantile forecasts
Pinball loss
Rolling-origin backtest
MASE
Atomic publish
Hierarchical reconciliation
Drift and retraining
Cold start

Functional requirements

Load daily sales per series, including late corrections, without double counting
Load known future inputs: holidays and planned promotions
Every night, forecast each series 28 days ahead as p10, p50 and p90
Publish a run only when it is complete and has passed checks; keep the previous run live until then
Serve the published forecast for a series, and export a full run in bulk
Backtest any model version on past data and report its accuracy
Track accuracy per series as actuals arrive, and flag series whose error has grown

Non-functional requirements

The run is published by 5:00 every day, an hour before orders are placed (a target we chose)
Readers never see a partial run: the switch from one run to the next is atomic
The forecast API answers in under 100 ms at p99 (our target; reads are simple key lookups)
A failed run leaves the previous run in place and alerts a person
Every published number can be traced to a run, a model version and the data cutoff it used

Interview questions

How would you size the nightly run?
Local models or a global model?
Why do you need quantiles, and how do you know they are right?
How do you backtest a forecasting model?
Why not use MAPE?
How do you make sure nobody reads a half-finished run?
How do you forecast a product that launched yesterday?
Forecasts at store level do not add up to the regional forecast. What do you do?