Call tester: how to test an AI phone agent with test calls

A call tester is anything that rings your phone line and checks what happens. It can be a person with a script or software playing a caller. To test an AI phone agent well, use both. Ring it yourself first, then rerun the same scripted calls after every change, then add simulated callers for volume. Mark each call against rules you wrote beforehand, and make enough calls that the pass rate means something.

Key facts

• Vapi's testing page says "one successful test call is not enough" (Vapi, Testing voice agents).
• Ten passes in ten calls still leaves a Wilson 95% interval of about 72% to 100% (NIST method).
• Ninety passes in 100 calls gives a Wilson 95% interval of about 83% to 95% (NIST method).
• Retell says a phone call also tests the carrier audio path, which text simulations skip (Retell, phone call testing).
• Vapi's simulations let an AI tester hold a conversation with your assistant, in chat mode or with full audio (Vapi, Simulations overview).

Vapi's own testing page explains why: because an agent can answer the same caller differently, one test call proves little.

What kinds of test call are there?

There are three, and each finds faults the others miss.

Manual calls are you or a colleague ringing the number from a real phone. Nothing beats them for hearing the delay, the voice and how the agent copes when you talk over it. They are slow, though. People also drift from the script without noticing.

Scripted callers follow a written script with a goal, the lines to say and the result you expect. A person can read it, or a recording can be played down the line. Because the call is the same each time, a fault found today can be checked again after the fix.

Simulated callers are a language model playing the caller from a short brief, such as a customer who wants to move a delivery and keeps changing the date. They can run many calls in a row and vary their wording each time. Several platforms offer this. In Vapi's simulations, an AI tester holds a conversation with your assistant, in chat mode or with full audio. Retell's simulation testing runs as a text conversation. Outside tools, RingTru among them, ring the agent over the phone with simulated callers.

Why do you still need real phone calls?

Text tests and in-platform simulations skip the phone network. Retell's phone call testing page says a phone call also tests the carrier audio path, and lists keypad menus, transfers, voicemail detection and SMS as things to check on a real number. Vapi's testing page says synthetic callers do not fully represent real callers and their environments. It points to manual calls and production recordings for background noise, degraded audio, accents, silences and interruptions.

So keep some real calls in every round, even when most of the volume comes from simulated callers. Our guide to text vs voice testing sets out which faults only show up in audio.

What should you check on each call?

Write the checks down before you call. A check you invent after hearing the call tends to pass. This list covers most agents.

• Outcome. Did the caller get what they rang for, and did the outside system really change? A booking the agent confirms but never makes is a fail.
• Facts. Did the agent say anything untrue or invent a policy? See hallucination.
• Speed. How long did the caller wait after they stopped talking? This is voice-to-voice latency. Time it on the recording rather than trusting a dashboard average.
• Turn-taking. Did the agent cut in while the caller paused, and did it stop when talked over? See barge-in and endpointing.
• Hearing. Did it catch names, numbers and postcodes first time?
• Handover. When the caller asked for a person, or should have been passed to one, did the handover happen, and did the person get the context?
• Ending. Did the call end cleanly after goodbye, and never in the middle of a task?
• Evidence. Is there a recording and a transcript you can go back to?

Give each check a plain pass or fail. A rubric written as simple questions keeps the marking consistent, whether a person marks the call or an AI judge does.

Which calls belong in the set?

Start with the calls that would hurt most if they went wrong, then add the awkward ones. A first set might be:

1. The main task, start to finish.
2. A caller who leaves out a detail the agent needs.
3. A caller who changes their mind halfway.
4. A caller who interrupts a long answer.
5. A caller who pauses while reading out a number.
6. A request the agent should refuse or pass to a person.
7. A transfer where nobody answers.
8. A noisy line, and a caller with a strong regional accent.
9. A caller who goes silent.
10. A call that reaches voicemail, for outbound agents.

For the hard cases, see identity checks, handover and looping failures and accents and background noise.

Keep each one as a scenario with its expected result. Every fault you fix becomes a new scenario. Together they form your regression set, which you rerun after each change to the prompt, the model or the voice.

How many test calls do you need?

More than most teams make. A pass rate from a few calls is a rough guess, and the honest way to show that is a confidence interval. For pass rates, use the Wilson interval. The NIST handbook describes the Wilson method and notes that, unlike the simple normal approximation, its lower limit cannot be negative.

The table shows what the Wilson formula gives at 95% confidence. These are sums, not test results.

Ten clean calls in a row still leave room for a true pass rate near 72%, by the Wilson method. That is why one good call proves so little. For a scenario that matters, such as an identity check, aim for 30 or more runs and report the count beside the percentage. Retell gives similar advice for its own simulations, which is to judge a scenario on its pass rate across runs and not on one run (Retell, simulation testing). Our Retell guide works through the same method.

Two cautions. Calls that reuse one script word for word test less than the count suggests, so say when repeats were identical. And a pass rate only covers the callers you tried, so a set with no accents or noise says nothing about them.

We have not published pass rates for any agent or platform yet. Our own benchmark figures are planned, and the benchmark method is public.

Where to go next

The voice agent testing guide covers the wider method. For the testing tools on a named platform, see the test guides by platform. If you are still choosing a platform, our Vapi vs Retell vs Bland comparison ends with how to decide by testing your own calls.

Everything here was checked against the linked pages on 28 October 2026.