Chaos + Observability

How chaos engineering and AIOps intersect — steady-state hypothesis definition, using AIOps to validate that fault injection produces expected alert groups, AI-guided experiment selection, Chaos Engineering 2.0 with service-mesh-native fault injection and policy-as-code gates. When chaos is premature and observability must come first.

Overview

Chaos engineering in one picture

Chaos engineering deliberately injects failures into a system to find weaknesses before they cause production incidents. Netflix popularized it with Chaos Monkey in 2011; the community has since turned it into a structured experimental method with prerequisites, hypothesis formats and success criteria.

The core principle

It is the scientific method applied to distributed systems: a falsifiable hypothesis, a controlled fault, a measurement and a lesson.

Steady-state metric — p99 latency below 200ms and error rate below 0.1%
Fault — Terminate one auth service pod
Expected outcome — Steady state is maintained

> [!NOTE] Full example: "We believe the system maintains p99 latency below 200ms and error rate below 0.1% if we terminate one auth service pod."

Observability comes first

Chaos without observability is not engineering; it is random damage. If you cannot measure whether steady state held, you learn nothing.

Where chaos meets AIOps

The two validate each other.

AIOps validates chaos — Does anomaly detection fire within the expected window after the fault? Does correlation group the alert storm into one incident?
Chaos validates AIOps — Given a known fault with known expected alerts, does AIOps detect, correlate and surface the right information?

Chaos engineering 2.0

The 2025 generation adds three capabilities, starting with a gate in front of every experiment.

Policy-as-code gates — Block experiments when SLO burn rate exceeds a threshold or a deployment is in progress
Service-mesh fault injection — Istio VirtualService httpFault injects latency and errors at the proxy, with no application code changes
AI-guided experiment selection — Analyzes service topology and past experiment history to recommend the highest-value next experiment by coverage gap

Interview questions

How would you design a GameDay for a team that has never run chaos engineering before?
How does Istio fault injection work and when should you use it instead of LitmusChaos?
How do you use AIOps tooling to evaluate the coverage quality of your monitoring after a chaos experiment?
What are the organizational and process prerequisites for running chaos in production?
How does AI-guided chaos experiment selection work and what data does it need?