Real-World MLOps Case Studies

Overview

Why case studies matter

Production MLOps rarely looks like the textbook diagram. Latency budgets, data freshness windows, organizational boundaries and cost ceilings end up driving the design. The three cases here span three clouds and three kinds of ML problem.

• Demand forecasting — Azure — Time-series forecast — Nightly batch endpoint — Challenger vs incumbent on validation — Drift during promotions
• Handwriting assistance — GCP — Computer vision classifier — Real-time Vertex AI Endpoint — 90/10 A/B, 48-hour window — AutoML ceiling on cursive
• Precision delivery — AWS — Real-time route scoring — SageMaker real-time endpoints, two regions — Two weeks in shadow mode — Feature staleness over 90 seconds

Case 1: demand forecasting on Azure

A retailer needed daily inventory predictions across thousands of SKUs and store locations. Azure Databricks does the feature engineering (lag features, rolling statistics, calendar encodings) and Azure ML handles training and experiment tracking.

• Event-driven retraining: a data freshness check triggers retraining when new sales data lands, not a fixed calendar, so promotions do not leave the model stale.
• Versioned features: Azure ML Datasets version every feature table, so pinning dataset and model versions reproduces any prediction.
• Drift alerts: PSI on the input sales distribution; above 0.2 an immediate retraining job runs instead of waiting for the weekly schedule.

> [!TIP] Forecasting degrades faster than classification under shift because small changes compound across a multi-step horizon. Setting drift thresholds by business impact (missed stock events) instead of p-values cut false-positive retraining runs by 60 percent.

Case 2: handwriting assistance on GCP

An edtech product scores children's letter formation in real time. The model is a computer vision classifier trained with Vertex AI AutoML Vision, plus a custom fine-tuning stage for language-specific character sets.

• One versioned artifact: Vertex AI Pipelines runs data validation, training, evaluation and deployment together.
• Endpoint-level A/B: traffic weights split 90/10; the challenger is promoted only after a 48-hour window on child-specific engagement metrics.
• No edge at launch: the model exceeded the memory budget of low-end Android tablets.
• Profile before optimizing: the budget was under 300ms end to end; the model call took 40ms and client-side image preprocessing was the bottleneck.

> [!NOTE] AutoML got the first model out fast but set a ceiling. The team later swapped the backbone for a custom EfficientNet-B2 to fix accuracy on cursive letters.

Case 3: real-time precision delivery on AWS

A logistics company routes drivers in real time from traffic, weather and package priority. The model scores candidate routes every few seconds for each active driver.

• Proactive autoscaling: SageMakerVariantInvocationsPerInstance scales out before the queue backs up, not after.
• Online and offline stores: the online store holds precomputed features (segment speeds, driver location embeddings) fed by Kinesis; the offline store holds history for training.
• Multi-region: active endpoints in two AWS regions behind Route 53 latency-based routing.
• Shadow first: the new model ran in parallel for two weeks before taking live traffic.

> [!WARNING] Traffic features older than 90 seconds visibly degraded routes. The team spent more engineering time on the Kinesis pipeline than on the model.

Patterns shared by all three

• Versioned artifacts for reproducibility.
• Staged rollout (A/B or shadow) before full promotion.
• Drift or freshness monitoring as a first-class operational concern.
• Separate paths: latency-sensitive online serving, throughput-sensitive offline training.

Managed vs custom stacks

• Managed (Vertex AI Pipelines, SageMaker Pipelines, Azure ML) — Standard patterns, small team, need speed — Constraints on custom runtimes and integrations
• Custom (Kubeflow, Airflow) — Unusual runtimes, cost tuning, any data source — The ML team carries the operational burden

Interview questions

• Walk through the end-to-end architecture for the Azure demand forecasting pipeline, including how retraining, versioning, and drift monitoring interact.
• What are the key trade-offs between using a managed MLOps platform like Vertex AI Pipelines versus a custom stack built on Kubeflow or Airflow?
• How should a team design a real-time feature pipeline for a sub-100ms latency ML system, and what did the logistics case study reveal about the hardest operational problems?
• What are the most common causes of production ML failures revealed by these case studies, and what monitoring and rollback strategies address them?

More MLOps & LLMOps topics

• Model Drift Detection
• Model Evaluation — Offline and Online
• Model Packaging and Formats
• Model Registry
• Model Serving
• Online and Continual Learning
• Reproducible ML Pipelines
• Responsible AI and Explainability