Open-Source AIOps

Building AIOps capability without vendor lock-in — Prometheus anomaly detector (Fourier + Prophet), Grafana three-band PromQL anomaly detection, Robusta enrichment playbooks for Kubernetes context, DoWhy causal graphs for RCA, and the CCF AIOps challenge reference implementations. Honest read on when DIY is worth the maintenance cost.

Overview

Open-source AIOps in one picture

Open-source AIOps means assembling capabilities from composable components instead of buying a bundled vendor platform. The stack has four layers, and each one feeds the next.

Why teams build it themselves

Data residency: telemetry cannot leave your infrastructure.
Custom metric semantics: vendor tools do not understand your business KPIs.
Cost: vendor AIOps pricing can be prohibitive at scale.
Ownership: the organization prefers to own and understand every layer.

The four layers

Collection and storage — Prometheus, Loki, Tempo (the Grafana stack), Alertmanager — Stores metrics, logs and traces; Alertmanager routes, groups, inhibits and silences alerts and integrates with PagerDuty, Slack and OpsGenie
Anomaly detection — Prometheus Anomaly Detector (Red Hat AICoE), a custom statsforecast or scikit-learn service, the Grafana ML plugin — Scores metrics for anomalies and exposes the scores back to Prometheus
Enrichment — Robusta — Attaches Kubernetes context to every alert before routing
Causal inference — DoWhy (Microsoft Research) — Goes beyond correlation-based RCA to actual causal analysis

Anomaly detection writes back to Prometheus

The Prometheus Anomaly Detector is a Python service that runs Fourier or Prophet models against Prometheus metrics. It writes anomaly scores back as new Prometheus metrics, so they show up in Grafana and can be queried in Alertmanager rules.

Prometheus Anomaly Detector — A Python service — Fourier or Prophet
Custom Python microservice — Your own service exposing /metrics — statsforecast or scikit-learn
Grafana ML plugin — None — Basic Holt-Winters seasonal detection in panels

Enrichment with Robusta

Robusta is a Kubernetes-native runbook automation platform. It intercepts Alertmanager webhooks and attaches pod logs, events, resource usage and recent deploys to every alert before routing it on.

> [!TIP] Robusta eliminates the first five minutes of every Kubernetes incident: the context-gathering step.

When DIY is and is not worth it

Vendor pricing is prohibitive for your data volume — Build
Compliance or data residency keeps telemetry inside your infrastructure — Build
Your metrics have non-standard semantics vendor tools misinterpret — Build
You have the ML engineering capability to own model retraining pipelines — Build
Datadog Watchdog or Dynatrace Davis is already bundled with your APM subscription — Buy (marginal cost is zero)
Your team lacks the Python and ML skills to manage model drift — Buy
Running the extra services costs more than the vendor option saves — Buy

Interview questions

Walk me through implementing Grafana three-band anomaly detection on Prometheus metrics.
How does Robusta work and what playbooks are most valuable to implement first?
When does DoWhy's causal inference add value over correlation-based RCA, and what are its practical limitations?
How do you decide between buying a vendor AIOps platform versus building with open-source tools?
How would you use open-source tools to build an alert enrichment pipeline that reduces investigation time?

Open-Source AIOps

Overview