AIOps Meets MLOps

MLOps governs the full lifecycle of machine learning models from training through deployment and retraining, while AIOps applies those same ML capabilities to automate IT operations tasks like anomaly detection and incident correlation. The two disciplines are mutually dependent: AIOps monitors the health of MLOps pipelines, and MLOps manages the models that power AIOps systems.

Overview

Two disciplines, one loop

AIOps and MLOps are often conflated, but they work at different levels of the stack, and each depends on the other.

Covers — The ML model lifecycle: data ingestion, feature engineering, experiment tracking, evaluation, staged deployment, accuracy monitoring, retraining triggers — ML applied to IT operations: anomaly detection on metrics, alert correlation and deduplication, automated RCA, capacity forecasting
Core question — Is the model still correct, and can we ship an updated one safely? — Can intelligent automation replace or augment human toil in ops?

> [!NOTE] The anomaly detector watching your infrastructure is itself a model, with a training pipeline, a deployment lifecycle and a drift problem.

Monitoring ML models in production

A model can degrade without throwing a single error. Three signals catch it.

Data drift — Production input features drift from the training distribution — A fraud model trained on pre-COVID spending loses precision as consumer behavior changes
Concept drift — The relationship between features and target changes, even with stable inputs — Every demand-forecasting model trained before the pandemic, in early 2020
Output confidence — Predictions that clustered near 0.9 and 0.1 start clustering near 0.5: the model is confused — Needs no ground truth labels and can be monitored in real time

Feature store monitoring

Feature stores (Feast, Tecton, Hopsworks) serve real-time features at inference. When a pipeline fails silently, the model gets null or outdated features, still returns predictions, raises no exception, and is simply wrong.

AIOps for training pipelines

GPU OOM — Kills the process without a meaningful error — GPU memory spike detection
Data loader deadlock — The job stalls indefinitely — Runtime anomaly: run takes more than 2x the historical median
Gradient explosion — NaN loss, and many loops keep running on NaN — Loss curve divergence: loss rises monotonically after epoch N

Tooling: MLflow (experiment tracking, metric logging), Prometheus and Grafana (infrastructure metrics), Weights and Biases alerts (training curve anomalies), Amazon SageMaker Model Monitor and Vertex AI Model Monitoring.

The CI/CD/CT/CM pipeline

MLOps extends CI/CD with two stages. CT closes the loop: drift found by CM triggers retraining, and CD ships the new model.

This maps directly onto the AIOps automation maturity model, whose top tier is self-healing: detect, diagnose, remediate and confirm resolution without paging anyone.

The shared vocabulary problem

"Monitoring" means three things, and a production ML system needs all three at once.

Model accuracy monitoring — MLOps
Infrastructure health monitoring — AIOps
Business KPI monitoring — Product analytics

> [!TIP] Name the three explicitly. Without shared conventions, teams talk past each other in incident retrospectives.

Interview questions

Explain the circular dependency between AIOps and MLOps and give a concrete example of each direction of the dependency.
A fraud detection model was deployed six months ago and is passing all infrastructure health checks, but the business team reports that fraud losses have increased. Walk through how you would diagnose whether this is a data drift or concept drift problem.
Describe three ways a training pipeline can fail silently and how AIOps monitoring would detect each.
What is continuous training in the MLOps model and what triggers it? How does it connect to the AIOps automation maturity model?