Internal ML Platforms: Golden Paths for Model Teams

Overview

The job

A company with ten model teams and no platform has ten ways to build a CUDA image, ten ways to ask for GPUs, and ten serving stacks that page ten different people at night. The platform team makes the right way the easy way: a golden path (or paved road) that takes a model from a notebook to production with as few tickets and decisions as possible, and a documented door for teams that need to go off-road.

A concrete case (illustrative). A marketplace company has eight model teams (search ranking, fraud, recommendations, an LLM support assistant) sharing two GPU clusters. Today a new model takes five weeks to reach production: two weeks waiting for GPUs, a week fighting CUDA versions, and a hand-built serving stack. The platform team of four ships, in order: two base image lines, a GPU queue with team quotas, Backstage templates for a training job and a model service, one reusable CI workflow with the evaluation gate, and KServe serving with canaries. Success is measured as time to first deploy, golden path share and GPU busy rate.

Interviews for ML platform and AI platform roles ask exactly this: what goes on the path, what stays with teams, how GPUs are shared fairly, and how you know it works. Pipeline stages are in *MLOps CI/CD Automation*, the registry in *Model Registry*, the portal itself in *Backstage*.

Interview questions

• You are the first platform engineer for eight model teams who each deploy differently. Design the golden path from notebook to production and the order you would build it in.
• Design the platform API that model teams use to deploy a model. Would you expose the serving runtime directly, a portal form, or your own custom resource, and how would you evolve it?
• A team says the golden path is too slow and wants to run its own serving stack. How do you respond?
• How do you manage CUDA, driver and framework versions across hundreds of model images without breaking teams?
• How would you measure whether an internal ML platform is succeeding, and what would you show leadership each quarter?

More MLOps & LLMOps topics

• Azure Machine Learning Platform
• Data Versioning and Dataset Management
• DevOps to MLOps Transition
• Feature Stores
• GitOps for Models: Versioned Rollouts with Eval Gates
• Hyperparameter Tuning and AutoML
• LLM Serving
• LLMOps
• ML Experiment Tracking
• ML Governance and Compliance
• ML Monitoring and Alerting
• ML Pipeline Orchestration