Model Registry and Artifact Store

Versions, approvals, rollout and rollback for ML models

Design a model registry: the catalog that records every trained model version with its lineage, stores its files once and immutably, gates what may reach production, and lets serving roll a new version out or back by moving a pointer.

Overview

A team trains a new fraud model that beats the current one offline. How does it reach production safely, and how does it come back out when it misbehaves? Without a registry, someone copies a file to a bucket, edits a config and hopes. Nobody can say which data trained the live model, who approved it, or what the previous one was.

A model registry is the catalog of trained models. A registered model is a name, such as fraud-scorer. Each training result registered under it is a numbered model version, linked to the run, code commit and dataset that produced it. MLflow's registry, for example, starts each model at version 1 and links every version to the run that produced it.

The key design move is to separate what never changes from what changes often. Versions and their files are immutable. What production serves is a small, mutable alias, such as champion, that points at one version. MLflow deprecated fixed stages (Staging, Production, Archived) as of version 2.9.0 in favor of aliases; its docs note that one version can carry several aliases, which makes A/B testing and rollout easier. Promotion and rollback then become pointer moves, not file copies.

Questions to ask first

How many models and how often? — About 200 models, 20 new versions a day
How big are artifacts? — Mostly 1 to 5 GB; a few large language model fine-tunes
Who reads the registry at serving time? — About 500 model-server instances, checking every 30 seconds
What must happen before production? — Automatic metric gates, plus a human approval for high-risk models
How fast must rollback be? — Under 60 seconds (a target we chose)
Must we answer 'what trained this model'? — Yes, for audits: run, commit, data version, approver

What you will be able to do

Treat model versions as immutable records and aliases as the only mutable pointers, and explain why.
Store large artifacts once by content hash, including adapters that share a base model.
Enforce gates and approvals before an alias may point at a version.
Roll a version out as a canary and roll it back in seconds with a pointer move.
Answer 'what trained the model that made this decision?' from lineage and the audit log.

Key concepts

Registered model
Model version
Alias
Content addressing
Lineage
Evaluation gate
Canary rollout
Rollback
Audit log
Model card

Functional requirements

Create a registered model and register numbered versions with run id, code commit, dataset version, framework, metrics and artifact hash
Upload and download artifacts of up to hundreds of gigabytes, stored once per unique content
Set, move and delete aliases (champion, challenger) with a reason, subject to gates and approvals
Serve a compact config per model: champion, challenger and challenger traffic share
Roll back by moving the champion alias to the previous version
Keep an append-only audit log of every registration, alias move and approval

Non-functional requirements

A registered version and its files never change; a delete is a soft delete with retention
Config reads available 99.99% of the time (a target we chose); if the registry is down, servers keep the last config they had
Rollback visible to all servers within 60 seconds (a target we chose)
Alias moves are atomic: two people promoting at once cannot both win silently
Every change is attributable to a person or pipeline

Interview questions

Why keep versions immutable and use aliases for production?
How do you make rollback fast and safe?
How do you store a 140 GB model and ten fine-tunes of it?
How does a model server learn about a new version without downtime?
How do you split traffic for a canary?
What gates would you put in front of production?
Two engineers promote at the same moment. What happens?