How to test a Vapi voice agent

To test a Vapi agent, use the testing tools Vapi gives you, then check what they cannot cover with real phone calls. Vapi's own docs say that "one successful test call is not enough" and suggest repeatable checks that you rerun after any change, as set out on its testing page. Start with text evals for decisions, move to simulated callers for whole conversations, and finish by listening to recordings and ringing the agent yourself.

Key facts

• Vapi's docs say "one successful test call is not enough" on its testing page.
• Vapi has two built-in test types, evals and simulations, per its simulations overview.
• Vapi says evals do not test speech recognition, audio quality or turn-taking.
• The Start Speaking Plan wait time defaults to 0.4 seconds, per the speech configuration page.
• The Stop Speaking Plan backoffSeconds setting defaults to 1 second, per the speech configuration page.

What does Vapi give you to test with?

Vapi has two built-in test types, and its docs say most production agents need both.

Evals are text checks run against a mock conversation, at a point you choose. The evals quickstart describes three kinds of judge. An exact judge checks a tool name and its arguments. A regex judge matches a pattern in the reply, such as a greeting wording or an ID format. An AI judge reads the last assistant message against pass and fail criteria that you write. They answer questions such as whether it asked for the missing detail before acting, and whether it called the right tool. You can run them from the dashboard or the API, and the same page says you poll for the result until the run has ended.

Simulations put an AI tester on the other end. The simulations overview says the tester holds a conversation with your assistant or squad, and each simulation pairs a scenario with a personality. Chat mode is faster and cheaper because it skips speech. Voice mode tests the model, transcriber and voice together. Latency limits apply only to voice runs. Vapi suggests chat for quick iteration and voice for final validation.

Two cautions from the simulations quickstart. First, its booking example is a conversation-only demonstration, so a pass shows what the agent said and not that an appointment exists in a calendar. Second, in a suite you build yourself, every tool without a mock runs for real, so mock or sandbox anything that books or sends. You must also add at least one structured output before a simulation can run.

Both tools target Vapi assistants and squads. If your agent runs elsewhere, they do not apply, and you would need an outside caller instead.

What do evals and simulations miss?

Vapi is open about the limits. Evals, it says, do not test speech recognition, audio quality or turn-taking. Voice simulations do not fully reflect real callers, and background noise, degraded audio, accents, silences and interruptions need manual calls or production recordings. A tool that reports success also does not prove that the outside system really changed.

That is where your own test calls earn their keep, and RingTru's simulated callers can ring a phone agent with accents, noise and interruptions to cover them. For why text checks alone fall short, see text vs audio testing. Our guides on phone audio, accents and noise and turn-taking and latency cover what to throw at the line.

Where do you see what happened?

The Calls tab in the dashboard, and the API, hold the evidence for each call. The call logs page lists:

• Transcript: speaker-labelled bubbles. Clicking a line jumps the audio to that moment.
• Recording: a waveform you can play and scrub, plus downloads that depend on the call and its artifact settings.
• Latency summary: time by stage, such as transcriber, endpointing, model and voice, with a per-turn table. The page says the values are averages and per-turn measurements, not percentiles.
• Logs: entries by category, including transcriber, model, voice, endpointing, tool and transport, downloadable as gzipped JSONL.

If a test call has no recording, check the recording settings first. On the recording page, an artifactPlan object controls whether the recording, detailed logs and transcript are kept, and the dashboard's Advanced tab shows the same switches. Detailed pipeline logging is also governed by the assistant's logging setting, which the call logs page describes.

How do Vapi webhooks help with testing?

Point the assistant's server URL at an endpoint you control and you can capture every test call automatically. The server events page lists the event types. Three are worth logging for testing.

• end-of-call-report carries the ended reason, the call object, and an artifact with the recording, transcript and messages.
• status-update reports the call status. For a phone transfer it is sent when the transfer starts, and the docs say it does not confirm that the destination rang or answered.
• user-interrupted fires when the caller interrupts the assistant, which makes barge-in easy to count.

Other events, such as tool-calls, transcript and transfer-update, are on the same page. Remember that transfer-update is sent for the older transfer path and not for every handoff, so do not rely on it alone.

Which settings need a real call?

These are the settings whose effect you can only judge by ear.

Starting to speak. The Start Speaking Plan on the speech configuration page has a wait time, which defaults to 0.4 seconds, plus smart endpointing options that work out whether the caller has truly finished. The docs suggest a longer wait when callers need time to finish their thoughts. Test with callers who pause to find a number or a date.

Stopping. The same speech configuration page describes the Stop Speaking Plan. Its numWords setting is how many words the caller must say before the agent stops, and higher values stop a bare "okay" from cutting the agent off. The voiceSeconds setting is how long the caller must speak, with a default of 0.2 seconds, and higher values reduce false triggers from background sound. The backoffSeconds setting is the pause before the agent resumes speaking, which defaults to 1 second. Try a real interruption, a quick "mm-hm", and a cough. Our interruption handling guide explains what each should do.

Transfers. The Transfer Call tool page says blind transfers connect the caller without speaking to the destination, and most warm modes speak a message or summary to the destination before connecting the caller. Its own advice is to place a test call, ask for the destination described in the tool, check the call log for the transfer, and use the same phone provider and destination as production.

Voicemail. On voicemail detection, the docs warn that setting the beep wait too low may cut your message off, and that aggressive timing needs testing against your own voicemail patterns. Call from a real mobile with voicemail on, and also answer as a human.

Ending the call. The Hang Up tool lets the model end the call, and the docs say it only does so when the prompt tells it when. Check that it ends after a goodbye and never mid-task. In a chat, the docs add, the tool does not end the session, so this one needs a phone or web call.

Which calls should you run?

Vapi's coverage guide suggests pairing success cases with restraint cases and ranking them by risk. A starter set might be:

1. The main goal, start to finish, such as a booking or a balance enquiry.
2. A caller who leaves out a required detail.
3. A caller who changes their mind halfway.
4. A tool or service that fails, to see whether the agent avoids a false confirmation.
5. A request that should be refused, or handed to a person.
6. A transfer to the real destination, and one where nobody answers.
7. An interruption early, mid and late in a long reply.
8. A pause in the middle of reading out a number.
9. An outbound call that reaches voicemail.
10. A noisy line and an accented caller.

Every real failure you fix should become a new test. The same coverage guide asks for exactly that, and it keeps the set honest as models and prompts change.

In short, Vapi's docs describe evals for decisions, simulations for outcomes, and calls for audio and carriers. For audio testing from outside the platform, a tool such as RingTru can ring the agent as a caller. See the other test guides by platform and the wider voice agent testing guide. If you are weighing a move, our guide to Vapi alternatives lists what to test before you switch, and Vapi vs Retell vs Bland compares the main options. To test a rival, see how to test a Retell AI voice agent.

Everything here was checked against the linked Vapi pages on 28 October 2026. Vapi's docs change often, so check the pages before you rely on a setting name or default.