What makes or breaks a phone voice agent

A good AI voice agent answers quickly, waits when the caller pauses and hears the details correctly. It also sticks to what the business actually knows, passes the call to a person when it should, and recovers when it gets something wrong. When that all works, the caller gets what they rang for and hardly notices the agent. Most failed calls trace back to the same six places, and each one can be tested.

Key facts

• A study of ten languages found a mean gap of about 208 ms between turns (Stivers and others).
• One study of OpenAI’s Whisper found about 1% of transcriptions contained whole phrases that were never said (Careless Whisper).
• In tau-Voice, GPT-5 with reasoning reached 85% in text, while voice agents managed 31 to 51% on clean audio (tau-Voice).
• The tau-Voice authors found 79 to 90% of failures stem from agent behaviour (tau-Voice).

What does a good call feel like?

A caller phones a dentist's booking line. The agent answers without a long gap, asks what they need, and lets them say "Thursday, no, Friday" without losing the thread. It reads the date back correctly, books it in the real diary, and says what happens next. If the caller asks for something outside its job, such as a complaint about a bill, it says so and gets a person on the line with the details already passed over.

None of that is clever. It is ordinary things done reliably, and the sections below take them in turn.

How fast should the agent reply?

Speed sets the feel of the whole call. People leave very little gap between turns. A study of ten languages found a mean gap of about 208 ms between turns in a set of 101 conversations. That is a reference for how human talk sounds, not a pass mark for an agent, but it explains why a long silence after the caller finishes feels wrong. Long gaps also make callers repeat themselves, and the repeat then collides with the agent's late answer.

To test it: time the gap from the end of the caller's last word to the start of the agent's first sound, on a real phone line. Report the median and the slowest results, and include turns where the agent has to look something up. The turn-taking and latency guide covers how.

Does it take turns properly?

Turn-taking has two halves. The agent should wait while a caller reads out a long number with pauses in it, and stop quickly when the caller cuts in. A short "mm-hm" should not stop it, and a real correction should. Get either half wrong and the call feels like a bad line, even when every word is right.

To test it: script callers who pause mid-number, say "hold on a second", talk over a long answer, and make small noises. Count the times the agent cuts in too early, and the times it keeps talking when it should have stopped. The interruption handling guide has the full list.

Can it hear the caller on a phone line?

Everything the agent does rests on the transcript of what it heard, and a wrong word is not flagged to anyone. Phone audio is narrower and noisier than a studio recording. Accents, traffic, a television in the background and a poor mobile signal all make it harder. One study of OpenAI's Whisper found that about 1% of transcriptions contained whole phrases that were never said, and 38% of those hallucinations included explicit harms. That study looked at one model, but the lesson carries over. Check what the agent writes down as well as what it says.

To test it: run the same script through different voices, accents, noise levels and a phone-quality codec. Score names, dates and reference numbers as exact matches, because most of the words being right is no use if the booking number is wrong. Add some stretches of silence and hold music. See phone audio, accents and noise and the UK accents guide.

Does it stay on task and use real data?

An agent can hear every word and still apply a rule wrongly, drift off its instructions or invent a policy that does not exist. The tau-Voice benchmark compared text and voice agents across 278 tasks. GPT-5 with reasoning reached 85% in text, while voice agents managed 31 to 51% on clean audio and 26 to 38% with noise and accents. The authors' analysis found that 79 to 90% of failures stem from agent behaviour. These are benchmark numbers on one task set, so read them as a sign of the gap and no more than that. They do not forecast your line.

To test it: ask questions the agent must answer from your real data, and questions it must decline. Change the facts behind the agent and check that the answer changes with them. Repeat each scenario many times, because one pass proves little. The tau-bench paper proposes a measure of reliability over multiple trials for exactly this reason. Callers who give a surprising answer, or change their mind halfway through a sentence, deserve their own cases.

Does it hand over to a person well?

Some calls should end with a human, such as a complaint, a distressed caller or a request the agent has no power to grant. A good handover starts when the caller asks or the situation calls for it, reaches someone who is actually there, and does not make the caller start again. A bad one drops the caller into an empty queue, a voicemail box or a loop of "let me transfer you" that never lands. A log entry is not enough, so confirm that someone really picked up.

To test it: ask for a person in plain words, in irritated words, and indirectly. Test what happens when the receiving line is busy, closed or unanswered. Check that the summary reaches the human and that the caller is not asked for everything again. The handover and looping guide goes deeper.

Can it recover from its own mistakes?

Every agent gets something wrong sometimes, so the question is what happens next. A good one reads details back, accepts a correction without argument, retries a failed lookup with an honest "one moment", and tells the caller plainly when something did not work instead of claiming it did. A caller who hears "that did not go through, let me try again" stays calm. One who is told it worked, when it did not, finds out later and rings back angry.

To test it: plant faults. Make a back-end lookup slow, make it fail, and have the caller correct a detail after the read-back. Then check both the words the agent said and the system it claimed to have updated. The identity checks guide shows this pattern for a case where recovery rules matter.

How do I put these together?

Testing the six one by one on clean recordings is not enough, because they interact. A slow lookup invites the caller to speak, noise makes the agent stop at the wrong moment, and a misheard date leads straight into a wrong booking. Build whole calls that mix them, run each many times, and keep the real audio so you can listen to the failures. RingTru’s simulated callers ring your agent with accents, noise and interruptions, and its AI judges score every call. The solutions pages walk through four kinds of call to test. For typed versus spoken tests, see text vs audio testing. The voice agent testing overview lays out a full plan, and the glossary explains any terms used above.

Findings here were checked against the linked sources on 27 October 2026. Benchmarks and models change quickly, so rerun your own tests when you change the agent.