LLM Observability: Traces, Spans & OpenTelemetry GenAI
Tracing LLM apps and agents with OpenTelemetry: the GenAI semantic conventions, span trees for model, tool and retrieval calls, token, cost and latency metrics, sampling, PII in traces, and joining traces to evals and feedback.
Overview
Why LLM apps need tracing
A slow or wrong answer from an LLM app is rarely one call. An agent turn can be a planning call, two tool calls, a retrieval, a second model call and a retry after a 429. Logs show each piece on its own line; a trace shows them as one tree with timings, so you can see which step was slow, which one failed and how many tokens the whole turn used.
Classic APM already gives you HTTP and database spans. What LLM work adds is a set of facts no generic span carries: which model was asked and which one answered, input and output tokens, cached tokens, why generation stopped, which tool was called, and, when you choose to capture it, the prompt and the reply.
> [!IMPORTANT] Observability for LLM apps is three jobs on one trace: debugging (why was this turn slow or wrong), accounting (tokens and dollars per user, feature and model) and quality (which traces got a thumbs-down or failed an eval).
Interview questions
- An agent turn took 14 seconds for one user. Walk through how you instrument the system so the next one is explainable in under a minute.
- Finance asks for LLM cost per customer and per feature, every day. How do you build it from telemetry?
- You run 50 million model calls a day. Design trace sampling that keeps every error and every slow turn without storing everything.
- Legal says prompts may contain personal data. How do you keep tracing useful without leaking it?
- Users press thumbs-down on bad answers. How do you turn that into something engineers can act on?
- The GenAI conventions are still Development status. How do you keep dashboards and alerts from breaking on every SDK upgrade?
Sign up free to read the full topic
More Agentic AI topics
- Embeddings & Vector Search
- Evaluating Agents: Task Suites, Trajectories & Regression Gates
- Evaluating RAG & Catching Hallucinations
- Fine-tuning LLMs: SFT, LoRA/QLoRA & Preference Tuning
- Guardrails & Prompt Injection Defense
- LLM Cost & Latency Engineering
- Model Context Protocol (MCP) Servers
- Multi-Agent Systems with LangGraph
- RAG in Production: Ingestion to Answer
- Real-time Voice Agents: STT → LLM → TTS
Start free on Cariara · Coding problems · System design studio