Guide · Intermediate · 10 min
Voice agents: how they work, platforms, latency, costs, and use cases
A voice agent listens, responds, and takes action over the phone or in an app. Compare architectures, latency, full operating costs, and practical safeguards.
A voice agent receives speech, prepares a response, and can take action in software through tools. It works over the phone, in a browser, or in an app, using either a pipeline of speech recognition, a language model, and speech synthesis, or a conversational audio model. Choosing one requires looking beyond the advertised per-minute price to conversational latency and the reliability of its actions.
What separates a voice agent from a synthetic voice
Speech synthesis turns text into sound. A voice agent also needs to listen, recognize when the other person has finished speaking, follow the conversation, and decide what to do next. A pleasant voice therefore guarantees neither a smooth conversation nor a correctly handled request.
The ability to act is another distinction. Answering a question about opening hours is conversational assistance; checking a calendar and booking an appointment requires a connection to a business system. Tool calling enables that action, but the server must check permissions and validate the parameters it receives.
Consider an illustrative appointment-booking scenario. The agent asks for the reason for the visit, checks availability, suggests a time, requests confirmation, and then records the booking. It announces success only after the scheduling software confirms it.
This separation prevents spoken claims from being confused with actual results. Saying an appointment is booked does not prove that a booking exists. The business system’s state must remain the source of truth, including after a call drops.
Before choosing a platform, define the service you need: providing information, collecting a request, carrying out an operation, or transferring the caller. Each requires different permissions, checks, and success criteria.
Two architectures: cascaded processing and conversational audio
A cascaded architecture first converts speech into text, sends that text to a language model, and then converts the model’s response into audio. Processing can be streamed: some components begin working before the entire message has arrived.
A conversational audio architecture handles audio input and output directly. It can preserve information that transcription loses, such as certain aspects of intonation. Its exact capabilities, available tools, and control options still depend on the model you choose.
| Criterion | Cascaded speech recognition, language model, and speech synthesis | Conversational audio model |
|---|---|---|
| Components | Separate, replaceable components | More integrated audio processing |
| Troubleshooting | Inspect transcripts and intermediate steps | Visibility depends on the events exposed |
| Response control | Text is available before speech synthesis | Control varies by interface |
| Optimization | Tune each component separately | Tune the model and its orchestration |
| Costs to examine | Audio, text, speech synthesis, and transport | Incoming audio, outgoing audio, and associated services |
A cascade makes it easier to pinpoint an error: incorrect transcription, a poor decision, or mispronunciation. It also creates more interfaces to monitor. An integrated audio model simplifies some connections without eliminating problems involving networks, turn-taking, or business operations.
The right choice depends on how well it handles your task. Collecting reference numbers demands exact reproduction; an educational conversation requires more flexibility. Compare both architectures using the same recordings and operations, with evaluation criteria defined before testing begins.
Platforms and frameworks to compare
Managed solutions provide infrastructure and configuration interfaces. Frameworks give developers more control while leaving them with more responsibility for running the system. Our AI tools selection links to the product profiles and their updated findings.
| Solution | Type | Starting point for evaluation | Work you still need to handle |
|---|---|---|---|
| Retell AI | Managed platform | Telephony, configuration, and call analysis | Business workflows, integrations, and validation |
| Vapi | Voice orchestration platform | Component assembly and connections | Provider selection and control of total costs |
| LiveKit Agents | Framework for real-time agents | Audio sessions, logic, and conversation management | Development and deployment choices |
| Pipecat | Python framework | Audio pipelines and modular components | Development, integration, and operations |
The official Retell AI pricing page lists features including transcripts, simulation testing, webhooks, and API access. Check which features your plan includes and which incur additional charges.
Vapi separates orchestration hosting from the components used. This breakdown helps when comparing configurations, but it also means examining several cost items.
The LiveKit Agents documentation describes a framework for building voice agents and their interactions. The official Pipecat repository presents a composable Python approach that connects AI services, transports, and conversational pipelines.
These descriptions are not a quality ranking. For each solution, request a demonstration using your scenario, an export of useful traces, and an estimate based on the same configuration.
Measure latency through to the sound the user hears
The useful measure of conversational latency runs from the actual end of the user’s speech to the start of an audible response. Measuring only the model or speech synthesis leaves out part of the wait the user experiences.
In a cascade, delays can come from end-of-speech detection, transcription, the model, a business-system request, and audio generation. Transport and playback buffers also contribute. Some processing overlaps, so adding up isolated component durations will not necessarily reproduce the total delay.
At a minimum, distinguish between:
- connection time before the initial greeting;
- the delay before a conversational response;
- the delay before a result that requires a tool;
- the time it takes to stop speaking when the user interrupts.
A brief spoken acknowledgment can help users understand that a search is underway. It should not conceal a stalled request or count as delivery of the result. Measure the first audible sound and the useful answer separately.
Keep the median and higher percentiles, with an explicit definition: the 95th percentile is the delay at or below which 95% of observations fall. Report the number of exchanges measured, the channel, the hosting region, and network conditions.
No universal threshold replaces testing. An agent checking a case file can explain why it needs time; a conversation practice exercise calls for more immediate exchanges. Set targets for each workflow, then monitor interruptions, repeated requests, and abandonment.
End-of-turn detection, interruptions, and conversation history
Detecting silence is not enough to establish that someone has finished speaking. They may be looking for a number, taking a breath, or hesitating over a date. Responding too soon cuts off the request; waiting too long makes the agent cumbersome to use.
The LiveKit documentation on turn-taking distinguishes voice activity detection, end-of-turn detection, and interruption handling. Configure these mechanisms together, then verify them in realistic conversations.
In particular, distinguish an explicit interruption from a brief acknowledgment. Stopping an explanation at every small sound can be as frustrating as continuing to talk after someone asks the agent to stop. Describe the expected behavior in your acceptance criteria.
Interruptions also affect conversation history. If a response is cut off before an essential detail is played, the system must not assume the user heard it. Where possible, distinguish between a generated response, transmitted audio, and content actually played.
For business operations, a spoken interruption does not guarantee cancellation of an action already submitted. The server must track its state and make that state available for checking. A booking initiated just before a disconnection must not be repeated without checking whether it succeeded.
Finally, test how conversations end: prolonged silence, refusal, a callback request, and a transfer request. Each situation should reach an understandable outcome without getting stuck in a loop of repeated prompts.
Calculate total costs beyond the advertised rate
A budget brings together several items: the platform, speech recognition, the language model, speech synthesis or conversational audio, telephony, phone numbers, storage, and monitoring. Some plans bundle these costs. Initial development and human handling of exceptions still need to be added.
As of October 24, 2026, Vapi’s official pricing lists approximately $0.05 per minute for hosting, plus components and any transport charges. That figure therefore does not represent the full cost of a conversation.
On the same date, Retell AI lists a range of approximately $0.07 to $0.31 per minute for its voice agents on the official page linked above. Its breakdown distinguishes voice infrastructure, voices, models, telephony, and optional features. Check the total for your configuration along with your carrier’s fees.
Billing units also differ. The Vapi cost documentation distinguishes audio minutes, model tokens, and characters used for speech synthesis. Converting an estimate into a per-minute cost therefore depends on assumptions about the conversation.
To prepare a budget:
- multiply billable durations by the applicable rates;
- add subscriptions, phone numbers, and optional features;
- include tests, failed calls, and retries;
- account for maintenance, monitoring, and human intervention;
- divide the total by the number of correctly resolved requests.
The LLM pricing calculator helps isolate the text-processing component. It does not replace a complete telephony cost calculation. Focus on cost per correct resolution: a cheap conversation that later requires a callback may simply move the expense elsewhere.
Telephony, browsers, and concurrent capacity
The channel is part of the product. A demonstration using a headset on a stable network does not validate a call from a mobile phone while traveling. Audio compression, network loss, speakerphone use, and background noise can all affect comprehension.
In a browser, account for microphone permissions and clearly show the current state: connecting, listening, processing, responding, or error. Provide another way to continue when the microphone is unavailable.
For telephony, check number routing, transfers, hold messages, and call termination. SIP is a signaling protocol used to connect telephone systems. A successful connection must also preserve the information needed for a transfer.
Concurrent capacity needs testing separately from monthly call volume. A service may receive relatively few calls overall but experience concentrated bursts of incoming calls. Observe rejected calls, queues, and changes in response times under those conditions.
Define fallback behavior before launch: transfer to an available team, voicemail, or an offer to arrange a callback. Check what happens if the model responds but the calendar fails, or if nobody answers a transfer to a person.
A partial outage must not produce a false confirmation. The agent should be able to explain that it has not completed the operation and pass along a request someone can act on, containing only the necessary data.
Suitable use cases and situations to exclude
The strongest candidates have a clear scope, an accessible business data source, and a fallback route. The scenarios below illustrate possible designs; they are not customer results.
| Use case | Possible action | Essential check | Fallback |
|---|---|---|---|
| Call reception | Identify the reason for calling and route the call | Verify the destination department | Switchboard or voicemail |
| Appointments | Check availability and book a time | Confirm the date, location, and person | Callback request |
| Case updates | Look up a status | Authenticate before disclosing information | Secure channel |
| Product support | Find a procedure | Check its relevance and source | Human adviser |
| Speaking practice | Simulate a conversation | Explain the limits of feedback | Text interface |
A simple task does not always need a generative conversation. A form may work better for entering a complex address; a phone menu may be enough to select a department. Include these options when defining the project.
Exclude operations from the pilot when an error would have consequences that are difficult to reverse. Fluent speech does not justify delegating a medical decision, a credit decision, or a complex contractual commitment without oversight.
Accessibility also requires alternatives. Some people cannot speak easily, have difficulty hearing, or use an unreliable connection. Voice should complement the service without making an essential process inaccessible.
The article on AI agents for customer service explores this operational perspective. Here, the central criterion remains whether the agent can correctly complete a clearly bounded request, with a person able to take over.
Personal data, identity, and permissions
Map data flows before deployment: audio, transcripts, model context, tool results, recordings, and logs. Identify who receives the data, where it is processed, and how long it is retained. An option that disables recording does not prove that every other form of processing is disabled.
France’s data protection authority, CNIL, explains the information required when calls are monitored or recorded, including the purpose, recipients, and people’s rights. Distinguish disclosure that an automated system is being used from disclosure about recording. Have the setup reviewed for your circumstances and the jurisdictions involved.
Treat neither a familiar voice nor the displayed phone number as sufficient authentication for a sensitive operation. Use a procedure appropriate to the risk, potentially through a channel where the user is already authenticated. Limit the information disclosed before that verification.
The caller’s words must remain data to process. They cannot change security rules or expand the agent’s permissions. Apply the same separation to documents and tool responses.
Permissions must be enforced on the server: permitted operations, accessible fields, parameter validation, and duplicate prevention. The guide to securing an AI agent explains these protections.
Finally, examine the logs themselves. A diagnostic system may retain information that never appears in the business application. Minimize what logs contain, control access, and test whether deletion mechanisms work in practice.
Build a pilot and decide based on results
Start with a narrow scope and a test system. You need a business owner, usable data, a fallback procedure, and acceptance criteria. The guide to evaluating an AI agent helps structure this validation.
- Describe the expected outcome. Specify the permitted actions, required confirmations, and situations that require a transfer.
- Prepare the conversations. Include ordinary requests, ambiguous wording, corrections, and refusals.
- Test the actual channel. Vary phones, speakers, pauses, and acoustic environments.
- Introduce failures. Make a tool unavailable, interrupt an operation, and simulate an unanswered transfer.
- Review the traces. Compare the request, the information understood, the action executed, and the result announced.
Track correct resolutions, critical errors, useful transfers, response times, and total costs. A transfer can count as a success when it protects the user. Conversely, a conversation that ends without a transfer may still leave the request unresolved.
Common problems
When the agent cuts people off, examine end-of-turn detection and background noise before changing models. When it seems slow, separate conversational waiting time, business-system requests, and audio playback. When it confirms an action that never happened, check the connection between the server result and the wording of the response.
Before expanding the pilot:
- Sensitive actions have server-side checks.
- Interruptions and disconnections do not create duplicates.
- Failures lead to an understandable outcome.
- Data and retention periods are documented.
- Results, response times, and costs meet the agreed criteria.
Keep a record of the evaluated configuration and rerun critical cases after every component change. A voice agent’s quality depends on the entire process, from the speech it receives to the result it verifies.
Related pages
Frequently asked questions
How does a voice agent differ from an interactive voice response system?
A conventional interactive voice response system routes calls using keypad selections or predefined intents. A voice agent can interpret requests expressed in everyday language and use business tools. A menu remains suitable when the choices are simple and the process needs to be fully predictable.
Can I keep my phone number when using a voice agent?
That depends on your carrier, your setup, and the connections the platform supports. Call forwarding or a connection between phone systems may let you keep your existing number. Also check the costs and what happens when a call is transferred to a person.
Can a voice agent run entirely on local infrastructure?
Yes, provided that audio transport and every processing component run within your infrastructure. Hosting only the orchestration program is not enough if it sends speech to external APIs. Assess hardware, maintenance, and French-language performance together.
Can a voice agent understand different French accents?
That depends on the system and the conditions of the call. Include speakers who represent your audience, using the phones and environments they would normally use. Pay particular attention to critical details such as names, dates, and case reference numbers.
Can a voice agent accept payments?
It can direct users to a payment process, but payment card details should not pass freely through the model's context or logs. Use a dedicated payment system, restrict permissions, and have the relevant teams approve the process.