LLM SRE Agents

How LLM-powered agents operate in SRE — autonomous hypothesis-testing loops, hypermodal AI combining predictive and causal reasoning, MCP tool calling over live telemetry, and the observability maturity required before any autonomy is safe. Covers Datadog Bits AI, Dynatrace Davis 2025, and PagerDuty Auto-Pause in depth.

Overview

From single inference to reasoning loops

LLM SRE agents are the 2024 to 2026 evolution of AIOps. Earlier systems made one pass over telemetry and produced an output (alert group, anomaly score, correlation candidate). Agents iterate: they generate hypotheses, call tools for evidence, evaluate it, update the hypotheses, and repeat until confidence is high or escalation is warranted.

The hypothesis-testing loop

When an incident fires, the agent receives the alert payload, affected services, current SLO status and recent change events.

Auth DB pool exhaustion — Query DB pool utilization
Recent deploy changed connection timeout — Fetch recent deploys for the auth service
Upstream latency propagating — Pull error traces from the last 30 minutes

Datadog reports two to three loops before conclusion in its Bits AI SRE case studies.

MCP and tool extensibility

Model Context Protocol (MCP) defines how an LLM calls tools without bespoke function signatures per vendor. Bits AI SRE uses MCP, so a new data source is added by defining an MCP tool, without rewriting the agent core.

Dynatrace hypermodal AI

Dynatrace coined "hypermodal AI" in 2025 for Davis combining three sequential AI modes.

The critical prerequisite

Agents amplify the quality of existing observability. An agent calling query_metrics() on a service with no metrics gets empty results and must infer, and those inferences look like evidence. Poor coverage does not stop output; it stops the output from being grounded in fact.

> [!IMPORTANT] Full metrics, traces and logs coverage for affected services is the prerequisite, not a nice-to-have.

Autonomous remediation (restart pod, roll back deploy) needs more guardrails on top:

Validated runbooks for each automated action.
Tested rollback paths.
Blast-radius estimation.
Human approval for any state-changing action until that action type has passed at least 20 approved executions.

Interview questions

How does the Datadog Bits AI SRE hypothesis loop work end to end?
Compare Dynatrace Davis hypermodal AI with pure LLM-based agents like Bits AI for incident investigation.
What guardrails are required before enabling autonomous remediation in an LLM SRE agent?
How do you evaluate LLM SRE agent outputs during a live P1 incident without adding cognitive load?
What is the 2026 state of LLM SRE agents -- what is production-ready and what is still experimental?