Real-time Voice Agents: STT → LLM → TTS

Overview

Why voice is a latency problem first

In a text chat a two-second wait is fine. In a spoken conversation it feels broken: studies of turn-taking across languages (Stivers et al., PNAS 2009) found the most common gap between one speaker stopping and the next starting is between 0 and 200 ms in every language studied, about 200 ms on average. A voice agent will not hit that, but every few hundred milliseconds it adds past a second makes callers talk over it, repeat themselves or hang up.

So a voice agent is designed around one number: voice-to-voice latency, the time from the moment the caller stops speaking to the moment they hear the first sound of the reply. Every stage streams, and every stage has a budget.

Interview questions

• Your voice agent takes 2.5 seconds to start answering. Walk through how you find the time and get it under a second.
• How do you implement endpointing so the agent neither cuts people off nor waits too long?
• Design barge-in handling for a cascade agent on phone calls through Twilio.
• Cascade or speech-to-speech model for a bank customer-support voice agent? Defend the choice.
• How do you measure end-to-end latency of a voice agent in production, not just in a demo?
• The LLM provider slows down and some turns take 6 seconds. How should the voice agent behave?

More Agentic AI topics

• Guardrails & Prompt Injection Defense
• LLM Cost & Latency Engineering
• LLM Observability: Traces, Spans & OpenTelemetry GenAI
• Model Context Protocol (MCP) Servers
• Multi-Agent Systems with LangGraph
• RAG in Production: Ingestion to Answer