Kubernetes AIOps

AIOps patterns specific to Kubernetes — kube-state-metrics for cluster health, Prometheus Operator ServiceMonitor pattern, Robusta enrichment playbooks, ML-driven autoscaling (VPA, HPA, KEDA), and cluster-level anomaly detection using pod churn rate, eviction pressure, and topology from owner references.

Overview

Kubernetes-native AIOps in one picture

Kubernetes-native AIOps applies ML-augmented operations to the Kubernetes control plane and workload layer. Cluster state and resource usage flow into Prometheus, get turned into features, and drive both anomaly detection and autoscaling.

Why Kubernetes is different

High metric cardinality — Every pod, node, deployment and namespace produces separate series
Ephemeral workloads — Pod lifetimes of minutes to hours break model training assumptions
Rich topology — Owner references (pod to ReplicaSet to Deployment to HPA) can be used for better anomaly attribution

The observability foundation

Two primitives underpin every Kubernetes AIOps pattern.

kube-state-metrics (KSM) — Watches the Kubernetes API and exposes object health as Prometheus metrics: pod phase, restart count, desired-versus-available gap, node condition, PVC status, HPA scaling pressure. Resource usage is cAdvisor's job, not KSM's
Prometheus Operator — Manages Prometheus through CRDs and auto-discovers scrape targets through ServiceMonitor objects, so nobody edits prometheus.yml for each new service

> [!NOTE] Together they give zero-touch observability: new services are scraped automatically and cluster state is exposed as metrics automatically.

Feature engineering

Raw Kubernetes metrics need transforming before ML models can use them.

Pod churn rate (new pods per minute relative to cluster size) — Leading indicator of CrashLoopBackOff storms
Eviction pressure on several nodes at once — Cluster-wide resource exhaustion, not one workload's problem
Owner reference chains (kube_pod_owner to kube_replicaset_owner to kube_deployment) — The Kubernetes equivalent of a service topology graph

The owner chain is what separates two very different incidents:

Autoscaling intelligence

HPA — Scales pod count reactively on current CPU or custom metrics
KEDA — Scales to zero and reacts to external sources: Kafka lag, queue depth, Prometheus queries including ML-computed forecasts
VPA — Right-sizes resource requests from observed usage, reducing overprovisioning

Interview questions

What Kubernetes metrics are most valuable for cluster-level anomaly detection and why?
Explain how to implement predictive autoscaling using KEDA and Prophet together.
How does Robusta reduce Kubernetes incident MTTR and what playbooks should you build first?
What are the key differences between Kubernetes AIOps and traditional VM-based AIOps?
How do you handle anomaly detection on Kubernetes metrics that have very high cardinality?