An AI voice agent is software that handles a live phone call by running three components in a real-time loop: a speech-to-text model converts what the caller says into text, a language model generates a reply, and a text-to-speech model speaks that reply back. The round trip from the last syllable the caller speaks to the first syllable the agent plays back takes 200 to 500 milliseconds on a well-tuned system.
That's the answer. The rest of this post explains what's inside each step and why it matters when you're building one.
I built my first voice agent for a French restaurant, Le Marquier, in 2024. Before I started, I read everything I could find about "what is an AI voice agent." Most of what I found was either a vendor's homepage or a listicle of use cases. Neither told me how the thing actually worked. I had to piece it together from documentation, vendor benchmarks, and a lot of calls that cut off mid-sentence until I fixed the VAD settings.
The thing I learned is that a voice agent is not a chatbot with a speaker attached. The real-time constraint changes every design decision. Understanding the pipeline is not academic background; it determines which tradeoffs you make and which problems you can diagnose when something goes wrong.
I've since built voice agents for a jeweller, an SAP order-entry workflow, and several inbound call flows. Each one taught me something different about where the pipeline breaks under load. I'll cover what I've learned in each section below.
The three-component pipeline
Every AI voice agent runs the same three-part loop, regardless of which platform you use. Here's what happens on each turn of the conversation:
1. Speech-to-text
The caller's audio arrives as a stream. Before any transcription happens, a voice activity detection algorithm watches the audio for silence. When it detects a pause (typically 200 to 500ms of quiet), it treats that as the end of a turn and sends the accumulated audio to the STT model.
This is the first place I'd look when a voice agent feels slow: the VAD timeout. A 500ms silence threshold means the agent won't start processing until the caller has been quiet for half a second. That's fine for most business calls. For a fast-paced exchange where callers give short answers, I tune it down to 250 to 300ms. Go too short, though, and the agent starts cutting off callers mid-thought. On Le Marquier, I landed at 350ms after testing with ten real callers who volunteered to try the system before launch.
Once the audio goes to the STT model, the model returns a text transcript. Streaming STT models (Deepgram Nova-3, AssemblyAI Universal-2, Speechmatics Agent STT) start returning partial transcripts before the full audio is processed. This lets the LLM step start earlier, which matters for total latency. On a non-streaming model, you wait for the complete transcript before sending anything to the LLM, which adds 100 to 200ms of wall-clock time.
STT accuracy matters more than I expected when I first started building. A word error rate of 5% sounds acceptable until you realize that "book a table for Friday" and "book a table for five" are semantically different and the model will sometimes get it wrong. In noisy environments, WER degrades quickly. The Krisp Voice Isolation benchmark published on 28 September 2026 tested 11 STT configurations on real audio with a second voice in the background, and found that word error rate dropped 73% when voice isolation was applied first. If your callers are often in loud environments, WER on realistic noisy audio (not clean studio recordings) is the metric to optimize.
Speechmatics launched Agent STT in September 2026 with a stated goal of catching "high-consequence errors, like a wrong digit or missed negation, that derail voice agents." I haven't benchmarked it myself yet, but wrong digits are exactly the failure mode that matters most in booking and order-entry workflows.
2. The language model
The transcript from the STT step goes into an LLM along with a system prompt and the conversation history. The LLM generates a text reply, usually token by token in a streaming fashion.
The system prompt is where I spend the most time on any voice agent build. It defines the agent's persona, what it's allowed to do, how it handles common edge cases, and what it should say when it doesn't know the answer. For Le Marquier, I built the system prompt through about twenty iterations. It needed to handle French and English, collect a name and party size, ask about dietary restrictions, confirm a booking time, and do all of it without ever sounding scripted. The prompt that shipped was 847 tokens.
For voice, I always stream the LLM output. The model starts generating tokens, and I pipe the first complete sentence fragment to the TTS step before the rest of the response is generated. Without this overlap, the caller would wait for the full reply to be written and then spoken. With overlap, the TTS starts playing the first sentence while the LLM is still writing the second one. That's what makes sub-400ms total round-trip latency achievable.
Time-to-first-token is the LLM metric that matters here, not throughput. A model that starts producing tokens in 100ms is more useful for voice than one that takes 400ms but then generates very quickly. In my latency benchmark, I measured time-to-first-token across several providers with a 500-token system prompt and a short user turn. The numbers ranged from about 80ms to over 400ms depending on the provider and time of day.
Tool calls fit here too. If the agent needs to look up a booking, check availability, or submit an order, the LLM emits a function call rather than a text response. The orchestration layer intercepts it, runs the function, and feeds the result back into the next LLM turn. The caller hears a brief filler phrase ("Let me check that for you, one moment") while the tool runs. I've found that filler phrases need to sound natural and vary slightly to avoid feeling mechanical. On a workflow that checks availability, I rotate between three filler phrases rather than using the same one every time.
Getting tool calls right is the hardest part of a complex voice agent. The filler needs to bridge the wait without the caller losing confidence, and the tool call needs to complete within about two seconds. If a database query or external API call takes longer than that, the agent needs a second filler phrase, which starts to feel awkward. I test every tool call under load before launch to get a realistic p95 response time.
3. Text-to-speech
The LLM's response text goes to a TTS model, which synthesizes audio. For voice agents, the metric I care about is time-to-first-byte: how quickly the TTS model starts producing audio after receiving the first sentence fragment.
Modern neural TTS has gotten fast. Cartesia Sonic, which I use on most of the agents I build, routinely hits TTFB under 50ms. Inworld shipped Realtime TTS-2 in late September 2026 with a Flash variant at 25ms TTFB. ElevenLabs and Play.ai are both competitive in the 50 to 80ms range. These numbers are fast enough that TTS no longer dominates the latency budget; it used to be the bottleneck and now it's just one contributor.
Voice quality has also improved dramatically. When I built the Le Marquier agent in 2024, the TTS voices available were noticeably synthetic. Current neural TTS models produce audio that callers frequently don't flag as robotic, provided the conversation flow is natural. The voice quality problem is largely solved; the latency and conversation design problems are what remain.
One TTS consideration I underestimated early: punctuation matters a lot to phrasing. The LLM generates text without speaker intent; the TTS model interprets commas and periods to determine pauses and stress. A sentence that looks fine in text can sound flat or oddly paced when spoken. I now run every response template through the TTS before deployment and adjust punctuation where the audio sounds wrong.
How it differs from IVR, chatbots, and human agents
These four technologies serve similar ends but work very differently. Here's what actually separates them:
| Dimension | IVR | Text chatbot | AI voice agent | Human agent |
|---|---|---|---|---|
| Input handling | Key presses, fixed words | Typed natural language | Spoken natural language | Spoken natural language |
| Response logic | Decision tree | LLM or scripted | LLM with tools | Human judgment |
| Latency tolerance | Any (deterministic) | Seconds acceptable | 200-500ms required | Natural pauses |
| Cost per call | Near zero | Near zero | $0.05-0.20 per minute | $1.50-4.00 per minute |
| Handles open-ended input | No | Yes | Yes | Yes |
| Operates after hours | Yes | Yes | Yes | Expensive or not at all |
The IVR comparison is the most common question I get from business owners considering a voice agent. IVRs are cheap, reliable, and widely deployed. Their problem is caller drop-off: when the caller says something outside the expected script, the IVR falls through to a "I didn't understand that, please try again" prompt. Callers tolerate one of those. Two or three and they hang up or start pressing zero for a human. A voice agent handles varied phrasing without requiring the caller to learn the menu.
The text chatbot comparison matters because I've seen people try to port a text chatbot to voice by adding a microphone and a speaker. It doesn't work. A 3-second response time on a text chatbot is unremarkable; people read at different speeds and they're doing other things. A 3-second silence on a phone call reads as a dropped connection. The real-time constraint changes every design decision: the system prompt, the response length, the tool call architecture, the filler phrase strategy. I've learned this the hard way on builds that started as chatbot ports and had to be substantially rearchitected.
Conversational state: how the agent tracks what was said
The LLM receives three things on each turn: the system prompt, the conversation history, and the latest user transcript. The history is the full conversation so far, formatted as alternating user and assistant turns.
For a short call (three to five turns), this is trivial. Everything fits in the context window with room to spare and I don't think about it. For a complex workflow with many turns, I have two choices:
Keep everything in context. The full conversation history stays in every LLM call. Simple to implement. Token cost grows linearly with call length. Works fine for business calls, which rarely exceed fifteen turns for structured workflows like booking or order collection.
Track state separately. I maintain a state object and update it with each turn. The object might look like this for a table booking flow:
{
"name": null,
"party_size": null,
"date": null,
"time_preference": null,
"dietary": [],
"confirmed": false
}I inject a summary of the current state into the system prompt rather than the full transcript history. This keeps token count stable regardless of call length and gives me structured output at the end that I can hand off to a booking system or show to a human reviewing the call.
I use the state-object approach for any workflow where I'm collecting more than five distinct fields or where the call might branch in multiple directions depending on earlier answers. For simpler flows (answer a FAQ, collect a name and number for callback), keeping everything in context is less work and works fine.
The state object also makes human escalation cleaner. When the agent reaches a situation it can't handle (a request outside its scope, a distressed caller, a complex complaint), it transfers the call to a human agent and passes the state object so the human starts with full context rather than asking the caller to repeat everything.
The latency budget
Total round-trip time from end-of-speech to first audio syllable is the number I track on every build. Here's where each millisecond goes on a typical configuration:
| Step | Typical range | What drives variance |
|---|---|---|
| VAD timeout | 200-500ms | Configured threshold; trade-off between responsiveness and false triggers |
| STT inference | 50-200ms | Streaming vs batch, model size, audio length |
| LLM time-to-first-token | 80-400ms | Provider, system prompt length, time of day / load |
| TTS time-to-first-byte | 25-100ms | Provider, sentence length; most providers now under 80ms |
| Network (telephony) | 20-80ms | Provider, region; usually small compared to model inference |
A conservative configuration hits 900ms to 1.2 seconds total. A well-tuned one hits 350 to 450ms. At 350ms, the agent feels like a fast human. At 1.2 seconds, it feels like a noticeable pause that some callers interpret as network trouble.
The variable I have the most direct control over is VAD timeout. I start every build at 400ms, then watch recordings for where callers get cut off or where the agent starts speaking over someone mid-sentence. I adjust from there in 50ms increments. In my experience, 300 to 350ms is the sweet spot for most business call types in English. For languages where sentences end more gradually (Hindi, Japanese), I go slightly longer.
For LLM latency, system prompt length matters more than most builders realize. A 1,500-token system prompt can add 100ms to time-to-first-token compared to a 500-token one, at the same temperature and model. I review system prompts for redundancy before optimizing the rest of the pipeline; cutting a third of the tokens often cuts 50ms. My vendor latency benchmark has the numbers by provider.
A dev.to guide published on 28 September 2026 covers building sub-400ms voice agents for enterprise telephony in detail, with specific VAD tuning, streaming STT configuration, and turn-level budget breakdown. I'd point any builder targeting sub-400ms at that guide alongside my benchmark.
When a voice agent works and when it doesn't
I've built enough of these to have a view on where they succeed reliably and where they struggle.
Voice agents work well when:
- The call type is repetitive. If 70% of your calls follow one of three paths (book, cancel, FAQ), you have a strong candidate. The LLM handles the variation; the state machine handles the flow.
- Call volume is high enough that per-call human cost is measurable. A business fielding 50 calls a day that each take five minutes has a human time cost I can calculate and compare against a voice agent's $0.05 to $0.20 per minute.
- After-hours coverage matters. A voice agent handles a call at 2am without requiring on-call staff. For businesses that lose bookings because nobody picks up after 6pm, this is the clearest ROI case.
- The caller input is mostly structured. Dates, times, names, party sizes, order items. The more the input looks like filling out a form, the better the agent performs.
Voice agents struggle when:
- Calls are genuinely unpredictable. An agent I built for a B2B sales workflow fell apart on calls where the prospect wanted to discuss contract terms rather than book a demo. The more open-ended the call type, the more the agent needs careful fallback design and a clear handoff path to a human.
- The caller base has low tolerance for any friction. Older callers in some segments expect warmth and relationship signaling that current TTS can approximate but not deliver authentically. I factor this into the go/no-go assessment for each build.
- Audio quality is consistently poor. Voice agents degrade faster than humans on noisy lines. STT accuracy drops; VAD fires on background noise instead of speech. If a meaningful share of callers are in vehicles or loud environments, I test on realistic noisy audio before committing to a deployment.
One misconception I hear often: that a voice agent needs to handle every call to be worthwhile. It doesn't. An agent that handles 60% of inbound calls and transfers the remaining 40% to a human with a state summary can halve the time a human agent spends on routine calls while handling after-hours volume on its own. The right framing is: what share of calls is this agent well-suited for, and what does that translate to in time saved?
Build it or deploy one
This post is about what a voice agent is, not which product to buy. But the "build or buy" question comes up in every conversation I have about this, so I'll give my short answer: if you want to understand how voice agents work and build one yourself, the managed platforms (Vapi, Retell, Bland.ai) handle the pipeline orchestration and let you focus on the system prompt, tools, and conversation design. If you want to deploy a voice agent for your business without the build project, there are products built on top of this infrastructure that are ready to configure rather than code.
My write-up on evaluating AI agents before launch covers how I assess a build before putting it on live calls: AI Agent Evals for Builders. The short version is: test on realistic noisy audio, test your worst-case tool call latency, and test the barge-in behavior specifically before going live.
For the broader picture on AI agents (voice agents are a subset of the agent category), the AI Agents hub has the other posts in this series. And if you want to understand how conversational context and memory actually work inside an agent, I'm covering that in detail in a follow-up post on context engineering.
If you're looking to deploy an agent for your business rather than build one, Voxdonna is what I built for that purpose.
Frequently Asked Questions
What does an AI voice agent actually do on a phone call?
It listens to what the caller says (via speech-to-text), generates a reply (via language model), and speaks it back (via text-to-speech). This loop repeats for every turn in the conversation. The full cycle from end-of-speech to first audio syllable takes 200 to 500 milliseconds on a well-configured system.
How is an AI voice agent different from a phone tree or IVR?
An IVR follows a fixed decision tree and matches exact inputs like key presses or single expected words. A voice agent uses a language model and handles natural language: the caller can say "I want to book a table for two on Friday evening" and the agent understands all of it without requiring a specific format. The IVR is deterministic; the voice agent is probabilistic.
What is the latency budget for an AI voice agent?
The round trip from end-of-speech to first audio syllable takes 200 to 500 milliseconds on a tuned setup. VAD timeout (200 to 500ms), STT inference (50 to 200ms), LLM time-to-first-token (80 to 400ms), and TTS time-to-first-byte (25 to 100ms) all contribute. Sub-400ms total is achievable with streaming STT, a fast LLM provider, and a modern neural TTS model.
How does an AI voice agent handle interruptions mid-speech?
The voice activity detection monitor runs continuously, even while the agent is speaking. When the caller speaks during agent playback, the VAD detects it and triggers a barge-in: the agent stops its audio output and begins processing the new turn. The implementation challenge is making the agent stop mid-sentence cleanly and respond to the new input without treating the previous interrupted turn as fully resolved.
What are the main failure modes of AI voice agents?
The most common are: VAD settings that cut off callers mid-sentence or wait too long before processing; STT errors on domain-specific vocabulary or noisy audio; LLM hallucination on factual questions (confirming a slot that doesn't exist); and tool calls that fail silently. Testing on realistic noisy audio rather than clean recordings catches most of these before launch.
Ready to Get Started?
Book a free 30-minute discovery call. I'll look at your current call volume and show you exactly what a voice agent can handle versus what needs a human.
