How to test a voice agent on accents and background noise

A voice agent accents and noise test runs one fixed script of caller turns through the agent in each accent your callers have, with recorded noise mixed in at set levels. The audio is squeezed down to phone quality before each call, and names, numbers and addresses are scored separately for each accent group. Done in that order, you learn which callers your agent serves well and which conditions to fix first.

Key facts

• Phone calls usually arrive as audio sampled at 8,000 Hz (IETF RFC 3551).
• Mozilla Common Voice is described as about 2,500 hours from over 50,000 people across 29 languages in the latest release its authors report (Ardila et al.).
• EdAcc holds almost 40 hours of video-call conversation in many varieties of English (Sanabria et al.).
• The MUSAN noise corpus is released under a CC BY 4.0 licence (OpenSLR 17).
• A 2026 paper says aggregate word error rate "masks the unequal cost of errors" (Haq et al.).

Signal-to-noise ratio is how much louder the speech is than the noise. This guide is about doing: building and running the test. For the research on why phone audio and accents raise errors, read how phone audio and UK accents affect speech recognition.

Start from your callers, not a dataset

The best accent mix is a copy of your own caller base. Look at your call logs, CRM or sign-up data and count where callers come from, which languages they speak at home, and which regions they ring from. Write the mix down as a table of groups and shares. A dentist booking line in one city and a parcel line serving a whole country need very different tables.

Give each group enough calls to say something, because a group with a handful of speakers produces numbers that jump around from run to run. Also keep a few groups that are rare among your callers but costly to get wrong. A small share of callers is still a real group of people.

For UK regional detail and what studies found, see the UK accents and phone audio guide. This guide works for any mix, including one that spans several countries.

Where can you find labelled accent speech?

Public datasets give you real human voices with accent labels, which is a good start before you pay for recordings.

• Mozilla Common Voice is a crowdsourced corpus, described in its paper as about 2,500 hours from over 50,000 people across 29 languages in the most recent release the authors report. The TensorFlow Datasets entry lists an accent field for each clip. Its English option list in the project's code includes Indian and South Asian, Singaporean, Southern African, Filipino and Irish English, among others, though a comment there says the list is no longer the canonical one. Check how many clips carry a label for your groups before you rely on it.
• EdAcc, the Edinburgh corpus, holds almost 40 hours of video-call conversation between friends in a wide range of first and second-language varieties of English, under a CC-BY-SA licence. It is conversational, which is closer to a real call than read-aloud sentences.

Dataset labels are rough. A single "Indian English" tag covers a great many speakers, and a label tells you where someone says they are from, not how they talk. Treat these sets as a starting library, and add your own consented recordings of real callers' accents where you can.

How do you add noise at a set level?

Real calls happen in kitchens, cars and shopfronts. To test that, mix recorded noise into the clean speech. The key control is the signal-to-noise ratio (SNR), measured in decibels. Torchaudio's add_noise function shows the idea: it scales a noise signal and adds it to a waveform so the result has the SNR you ask for. Because SNR is set per clip, you can run the same script at several levels and watch where the agent starts to fail.

Public noise sets to use:

• MUSAN from Snyder, Chen and Povey is a corpus of music, speech and noise under a CC BY 4.0 licence. Its authors built it for voice activity detection and music or speech discrimination. Besides noise, it holds music and speech, which can stand in for a radio or a TV in the background.
• DEMAND is a set of environmental recordings made with a 16-channel array, and its file names suggest settings such as a kitchen, cafeteria, bus, car and metro. The page offers recordings at 48 kHz and 16 kHz, so downsample to your target rate before you mix. Its record count and licence wording are not consistent, so check the licence before you reuse the files.

Choose a ladder of levels, from nearly clean to nearly unintelligible, and always include a clean run as the baseline. Pick noise types that match your callers: traffic for a mobile line, a television for a home line, a busy room for a front desk. Do not assume a result for one engine carries over to another.

Make it sound like a phone call

Phone calls usually arrive as 8 kHz G.711 audio, and the UK accents guide explains why that matters. For the test, the order is what counts. Apply the phone step after mixing, not before. Noise is part of what the caller's handset or room picks up, so it goes in first, and the carrier then compresses everything together. Downsample to the 8,000 Hz phone rate, encode and decode through G.711, and only then feed the file to your agent. If you can, also place real calls over your actual carrier route, because carriers may transcode in ways a script does not copy. See phone audio, accents and noise for more background.

What should you measure?

Word error rate (WER), the share of words a transcript gets wrong, is the standard score for speech recognition. It is not enough on its own. A 2026 paper on British accents in financial voice assistants says aggregate WER "masks the unequal cost of errors", because errors in task-critical words such as merchant names, amounts and dates can change the action the agent takes.

Score these separately:

• Key-detail accuracy. Did the agent capture each name, number, postcode or date exactly right? Mark each item right or wrong.
• WER on the whole transcript. Still useful as a smoke test, and for spotting a change between two runs.
• Task success. Did the call end with the right booking, the right balance or the right handover? A good transcript can still produce a wrong action.
• Re-prompts and repeats. How often did the agent ask the caller to say it again, and how many turns did the call take? Repeats show friction that a transcript score can miss.
• Read-back corrections. When the agent repeats a detail and the caller says no, does it accept the fix?

Keep the audio, transcript and agent action for every run, so a bad result can be traced.

Report fairly by group

An overall average hides the callers who are served worst. Always report each accent group on its own line, with the number of speakers and calls behind it. Put the clean result and the noisy, phone-quality result side by side for each group, so readers see both the gap between groups and the gap between conditions.

The same paper reports accent-related failures "varying substantially across models and acoustic conditions". So avoid lumping several regional accents into one category, and never carry a finding over to a new engine, model version or carrier route. Rerun the set when any of them change.

Be plain about limits. If a group has few speakers, say so and show the spread of results, not just one number. If your test voices are synthetic, say that too, since a text-to-speech voice is not a stand-in for a person.

A short plan

• Write the caller mix, with groups and shares.
• Write one script heavy with names, numbers and addresses.
• Collect or record that script for each group.
• Make a clean copy and noisy copies at a ladder of SNR levels.
• Convert every copy to phone-rate G.711 audio.
• Run the agent, then score key details, task success and re-prompts by group.
• Fix the worst cell first, then rerun the whole grid.

Sources were read on 27 October 2026. Figures belong to the papers and data sets, not to your agent. Related: voice agent testing and interruption handling. For how many calls to run, see the call tester guide. RingTru runs this kind of test with simulated callers (see also accents and noisy lines), and our UK accents benchmark method shows how we score it.