Text vs voice testing for voice agents

Typed tests are enough for checking what the agent decides: its logic, prompt, policy and tool calls. You need audio the moment a result depends on hearing, speaking or timing. A good test set runs most checks as text, because that is quick and cheap, and repeats the speech-sensitive ones over audio before anything ships.

Key facts

• Pipecat recommends text mode for "fast, cheap iteration on prompts, logic, and function calling" (Pipecat).
• Vapi says its Evals "don’t test speech recognition, audio quality, or turn-taking" (Vapi).
• In tau-Voice, a text model reached 85% task completion while voice agents scored 31 to 51% on clean audio and 26 to 38% under realistic conditions (tau-Voice).
• A speech-to-speech agent "is evaluated in audio mode only" in Pipecat (Pipecat).

Why should I start with text tests?

Text tests skip the speech layers. The test types what a caller would say, the agent's language model replies, and nothing is transcribed or spoken. Pipecat's evals page describes its text mode as sent "as text, bypassing the STT" and recommends it for "fast, cheap iteration on prompts, logic, and function calling" (Pipecat). LiveKit says much the same: text mode is "a cost-efficient and deterministic way to test agent behavior" (LiveKit).

That speed changes the work. Hundreds of scenarios can run on every change, a new prompt can be tried in minutes, and a regression set can grow with each bug. Text suits questions like these:

• Does the agent follow the policy when the caller pushes back?
• Does it call the right tool with the right arguments?
• Does it recover when a tool returns an error?
• Does it stay on script across a long, branching conversation?

The typed words always arrive intact, so text results also stay steady from one run to the next. When a text test fails, the fault is in the agent's reasoning and not in what it heard.

What can typed tests not catch?

A typed test never lets the agent mishear anything. The speech-to-text step is gone, and so are the text-to-speech step and the clock. Pipecat's own docs say audio mode finds "problems that only show up with real speech: turn detection, words that sound alike, barge-in" (Pipecat).

Specifically, text cannot show you:

• Mishearing. A caller spells a surname or reads out a reference number and the recogniser gets one letter wrong. Text delivers the correct value every time.
• Accents and noise. Phone lines, traffic, speakerphones and strong accents change what the recogniser returns. See phone audio, accents and noise.
• Turn-taking and interruptions. With no audio stream there is nothing to cut in on and no pause to misread. See turn-taking and latency and interruption handling.
• Delay. Text has no recogniser, no voice synthesis and no network, so the pauses a caller actually feels never appear.
• Pronunciation. The agent's text can be right while the spoken version is unclear, such as a number read as one long string or a name said badly.

Vendor docs draw the same line. Vapi says its Evals "run at the text and model layer" and "don't test speech recognition, audio quality, or turn-taking", and points to controlled real calls for the phone path and live integrations (Vapi).

What the research shows

One public benchmark measures the gap directly. The tau-Voice paper benchmarks full-duplex voice agents and compares voice with text performance. In its results a text model with reasoning reached 85% task completion, while the voice agents scored 31 to 51% on clean audio and 26 to 38% under realistic conditions (tau-Voice).

Read that with care. The text baseline comes from a different setup than the voice agents, so the gap mixes model differences with the effect of voice. The paper also tests only audio-native models and leaves a cascaded speech-to-text, language model, text-to-speech setup for future work, and it runs its telephone audio at 8 kHz (tau-Voice).

Two findings are still useful. The authors say the gap "is not purely a speech recognition problem", because logic errors and made-up answers show up even on clean audio. And agents failed to transcribe names and emails even when they were spelled out letter by letter (tau-Voice). So voice testing turns up speech faults and also reasoning faults that only appear in a voice setting.

A plain rule for choosing

Ask what the scenario depends on. If it is only what the agent should decide or do, text is enough. If it includes anything the caller says out loud, or how it sounds or when it arrives, you need audio too.

Any scenario where the agent captures or reads back a value should exist in both forms. The text version proves the logic is right. The audio version proves the agent can hear the value and say it clearly.

There is one exception. If your agent is a speech-to-speech model with no separate text step, there is no text mode to run. Pipecat's docs say such an agent "is evaluated in audio mode only" (Pipecat). For design trade-offs, see speech-to-speech or a chained pipeline.

How should I mix text and audio tests?

Think in three layers, with most of the volume at the bottom.

1. Text on every change. Run the full scenario set in CI. It is fast enough to block a bad prompt before it merges.
2. Audio before release. Run the scenarios that involve hearing, speaking or timing, plus a sample of the rest. LiveKit's advice is to "reserve audio runs for turn-taking and speech-specific issues" (LiveKit). Pipecat pairs simulated callers with audio as "the pre-ship pass" (Pipecat).
3. A few real phone calls. Use controlled calls over the actual line for what only the line shows: codec effects, transfers, touch-tone digits and live integrations. Vapi recommends exactly this after its simulations (Vapi).

There is no magic ratio, and any split you see quoted is a starting guess. Begin with most scenarios in text, put every speech-sensitive one into audio, and adjust by comparing results. The habit that pays most is a paired set. Run the same scenarios in both modes and list where the outcome differs, because each difference is a fault that text alone would have hidden.

What the main platforms offer

Features change often, so check the linked page before relying on any of them.

• Pipecat has a text mode and an audio mode in its evals. See Evals overview.
• LiveKit runs unit tests in text. Its simulations talk to the agent in text by default and "can also run over audio to exercise the full STT-LLM-TTS pipeline". See Testing and evaluation. Its session events include interruption and false-interruption events, documented under Turns and interruptions.
• Vapi separates text-layer Evals from Simulations, which cover whole conversations. See Evals quickstart.
• Retell AI describes its simulation testing as running "as a text conversation" without placing a call, and lists live web or phone call testing as a separate option. See Automatically test your agent.

For setup detail on each, see test guides by platform.

Common mistakes

• Treating a green text run as proof the agent works on a call. It proves the logic and nothing else.
• Testing audio only with clean studio recordings. Add the noise and line quality your callers really have.
• Running every scenario in audio, then skipping runs because of cost. Keep audio for what needs it.
• Passing a read-back check on the written text alone. Listen to, or transcribe, what was actually spoken.
• Forgetting that a fix in the prompt can change spoken behaviour, so re-run the audio set after prompt changes too.

To start, take your ten most important scenarios and run them in text, then rerun the ones that capture a name, number or date over audio. The differences are your first real test plan. Our guide to test calls covers how many audio runs you need before a pass rate means anything. For the audio layer, RingTru’s simulated callers ring your agent with accents, noise and interruptions. For the checklist behind it, see what makes or breaks a voice agent. For the wider picture, read voice agent testing and the glossary.

Everything here was checked against the linked pages on 27 October 2026.