LLM as a judge for phone calls

An LLM as a judge is a language model that reads a record of a call and one written rule, then says whether the call met the rule and why. It works well on phone calls when each rule is specific, the judge quotes the line it relied on, and you check its marks against people.

Key facts

• The research that named the method found that strong judges reach "over 80% agreement, the same level of agreement between humans" (Zheng et al.).
• The same paper lists position, verbosity and self-enhancement bias, plus limited reasoning ability.
• ElevenLabs checks each criterion against the transcript and returns success, failure or unknown, with a rationale (ElevenLabs docs).
• Bland lets a judge read the transcript or listen to the audio, and puts it as "transcript reads what was said, audio hears how" (Bland docs).
• OpenAI advises that you calibrate automated grading against human marks before you rely on it.

What does a judge do with a phone call?

The idea comes from checking chat models. The authors of the original paper wanted to "explore using strong LLMs as judges to evaluate these models on more open-ended questions" (Zheng et al.), because people are slow and expensive to use at scale. A phone call is open-ended in the same way. A caller can confirm an address with "yes", "yeah, that's it" or "go on then", and a keyword check would miss two of the three.

So you give the judge three things. One is the call, as a transcript or as audio. Another is a single rule written in plain words, such as "before booking, the agent reads the address back and waits for a yes". The last is a request for a mark and the line that decided it. Our AI judges page shows one rule scored this way.

Should the judge read the transcript or listen to the audio?

Some judges read only the transcript. ElevenLabs says each criterion is "evaluated against the conversation transcript". That is cheap and easy to read back, but a transcript is only what the speech recogniser heard. If it misheard a postcode, the judge sees the wrong postcode and may pass or fail the call for the wrong reason. Our guide to UK accents on phone audio explains why that happens.

A transcript is text, so it carries no tone and may show little of the pauses or of people talking over each other. Bland's docs offer both modes, and say a judge in audio mode listens to the call, where transcript mode only reads it. Google says its Gemini models can analyse audio input directly, and lists emotion detection and speaker identification among the uses. We have not tested how well any audio judge does on phone-line audio.

A practical split follows. Use transcript judges for what was said, such as the right question asked or the wrong promise made. Use audio, or a timing check from the call log, for how it sounded, such as a long gap before the reply or an interruption that was ignored. Our guide to text and audio testing goes through which faults need which.

How do you write a rule a judge can score?

Write it so a person could make the same call from the same recording. A set of rules like this is called a rubric. OpenAI's evaluation guide says to "ensure eval rubrics are clear and detailed", and these habits follow from that.

• One rule per judge. "The agent was polite and accurate" is two rules, and when it fails you will not know which one broke.
• State what counts as a pass. "Handles the address well" gives the judge nothing to hold. "Reads the full address back and waits for the caller to confirm before booking" does.
• Allow a third answer. ElevenLabs lets a criterion return success, failure or unknown. A call that never reached the address step should not be marked as a failure.
• Prefer pass or fail to a score out of ten. OpenAI recommends pairwise comparison or pass/fail "for more reliability" (OpenAI).
• Ask for the quote and the time in the call first, then the verdict. OpenAI says that reasoning before scoring improves results, and a quote lets you check the mark in seconds.

Keep arithmetic and exact matching out of the judge. The original paper names "limited reasoning ability" as a weakness (Zheng et al.). Whether the agent sent the right account number to a tool is better checked by code that reads the tool call.

How does a judge get a call wrong?

The paper names three habits, and each has a phone-call version.

• Verbosity. OpenAI warns that "LLMs bias towards longer responses in general" (OpenAI). An agent that gives a long, friendly answer may score better than one that gives the right short one. Write rules about the outcome, not the tone, and read a few long passes yourself.
• Self-enhancement. The paper describes judges that may favour answers generated by themselves (Zheng et al.). If your agent runs on one model, judge it with a different one where you can.
• Position. The paper defines this as a judge favouring certain positions over others (Zheng et al.). It matters if you compare two agents or two versions on the same call. Swap the order and run it again.

Run the same call through the same judge a few times. A rule that gets a different mark on each run needs rewording before you trust it.

How do you check a judge against people?

OpenAI lists "not calibrating your automated metrics against human evals" as an anti-pattern. It says to scale up once the judge is "faster, cheaper, and consistently agrees with human annotations".

The steps are short. Pick a sample of calls that includes passes, failures and awkward cases. Have two people mark them without seeing the judge. Compare the people with each other first, because the paper's 80% figure is quoted against the level at which humans agree with each other. A judge cannot be held to a higher standard than your reviewers can reach. Then compare the judge with the people, rule by rule. A rule where it disagrees often needs rewording, or a person to check every mark.

Read the disagreements, not just the percentage. They usually point at one cause, such as a vague rule or a transcript error.

How we use judges at RingTru

RingTru's judges are designed around the pattern above. Each scores one written rule per call, quotes the transcript and gives the time in the call, and you can overturn a mark with a note. Our simulated callers are built to place the calls the judges then score. We say on the AI judges page that judges can be wrong, and that we will publish how often they agree with people before we quote any number.

Where we have not tested yet

We have not published an agreement rate between our judges and people, and this guide does not claim one. Nor have we compared transcript judges with audio judges on real phone-line recordings, or measured how often repeat runs of the same judge disagree. The figures above come from the linked sources and are not RingTru results.

Every source here was read on 31 October 2026. These guides name vendors in plain text and are not endorsed by any of them.