Evaluating Agents: Task Suites, Trajectories & Regression Gates

Overview

Why agents need their own evals

Evaluating a single LLM answer is comparing one output to a reference. An agent is different: it takes many steps, calls tools, changes the state of an environment and may reach a correct result by a wasteful or unsafe path. Two runs of the same agent on the same task can take different paths and get different results.

So an agent eval answers three questions at once. Did the task end in the right state? Was the path acceptable (no forbidden actions, a sensible number of steps)? And how reliably does that happen, at what cost, across repeated runs?

> [!IMPORTANT] A single run on a single task tells you almost nothing about an agent. Reliability is a distribution, and you only see it by running each task several times.

Interview questions

• Design an evaluation suite for a coding agent that edits a repository and opens a pull request.
• Your agent scored 62% yesterday and 57% today with no code change. What is going on, and how do you make scores trustworthy?
• Explain pass@k versus pass^k and when each is the right metric.
• Set up a regression gate in CI for prompt, tool and model changes.
• A new model version is available. How do you decide whether to upgrade your agent?
• How do you keep an agent eval suite from being contaminated or overfit?

More Agentic AI topics

• Browser & Computer-use Agents
• Context Engineering & Prompt Design
• Context Window Management for Persistent Agents
• Designing Tools a Model Calls Well
• Distillation & Small Models
• Embeddings & Vector Search
• Evaluating RAG & Catching Hallucinations
• Fine-tuning LLMs: SFT, LoRA/QLoRA & Preference Tuning
• Guardrails & Prompt Injection Defense
• LLM Cost & Latency Engineering
• LLM Observability: Traces, Spans & OpenTelemetry GenAI
• Model Context Protocol (MCP) Servers