GitOps for Models: Versioned Rollouts with Eval Gates

Overview

The job

A model platform team runs dozens of models, each retrained or fine-tuned every few weeks. Every new version is a production change with two failure modes code does not have: it can be slower on the same GPUs, and it can be worse while every health check stays green. An HTTP 200 with a wrong answer is still an outage for the product.

GitOps handles the operational half: the version each environment serves is one line in Git, Argo CD or Flux makes the cluster match it, and every change is a reviewed pull request. The model half needs more: eval gates before merge, a canary judged on quality as well as latency, and a plan for 100 GB of weights that must be on a GPU node before a pod is ready.

A concrete case. A customer-support SaaS company serves a fine-tuned 8B model on vLLM. Training registers version 42 in the model registry; a bot opens a one-line PR against the staging overlay; CI serves that exact artifact on an H100 runner and runs the golden, JSON, safety and latency suites; staging merges on green; a second PR with the same digest goes to production, where the model owner approves it and Argo Rollouts walks it from 5% to 100%, comparing canary with stable at each step.

What lands in Git is a reference, never the weights and never a moving name:

Interview questions

• Design the release path for 40 production models from the registry to the cluster, using GitOps. Who changes what, and where are the gates?
• Write the canary analysis for an LLM chat service. Which metrics, which thresholds, how long at each step, and how do you avoid deciding on noise?
• A model passed its canary and has served 100% for two hours when support tickets show it gives wrong refund amounts. Roll it back, and keep the system consistent.
• Roll out a new 70B model version across 16 GPU replicas. The weights are 140 GB and spare capacity is two replicas. Plan it.
• Design the offline eval gate that runs on a promotion PR. How do you keep it fast, stable and trusted?
• An auditor asks which model version answered a specific customer request last March, and who approved it. How does your design answer that?

More MLOps & LLMOps topics

• Agent Loop Prevention
• Agent Tool Security
• Azure Machine Learning Platform
• Data Versioning and Dataset Management
• DevOps to MLOps Transition
• Feature Stores
• Hyperparameter Tuning and AutoML
• Internal ML Platforms: Golden Paths for Model Teams
• LLM Serving
• LLMOps
• ML Experiment Tracking
• ML Governance and Compliance