You cannot evaluate a Live voice agent on isolated utterances, because Full-Duplex-Bench-v3 shows spoken language models still fail on real human audio – disfluencies, interruptions and chained tool use – unless those events sit in a held-out set.
That is not a vendor slogan looking for a paper. It is what the Full-Duplex-Bench line just made measurable. v3 is a spoken-language-model exam built from real human recordings, annotated for five kinds of mess people actually produce, then paired with scenarios that need chained API calls. The models on the table are the ones product teams are already shipping or shopping: GPT-Realtime, Gemini Live, Grok, Ultravox, and a cascaded Whisper-LLM-TTS stack. The failures that survive every architecture are the ones a booth script never contains. Self-correction. Interruptions. Multi-step tools under hard constraints. If your gold set is still isolated utterances, you are scoring a product you are not shipping.
The market pitch for Live agents has been latency and “it feels like a call.” Full-Duplex-Bench-v3 is the first widely cited exam that treats the call as a job: listen through the ums, update the slot when the speaker changes their mind, then actually invoke the tools in order. A transcription WER on a clean clip does not tell you whether the booking went to Boston after the speaker said New York first. A preference Elo does not tell you either. This bench is closer to production than either of those, and the scores are still ugly.
What v3 actually put on the table
Earlier Full-Duplex-Bench versions asked whether a model could take turns and handle overlap. Useful. Incomplete. Tool-use boards that already existed for voice either synthesised the audio or stayed on single-step calls, which is how you get a clean graph and a product that dies when a person talks like a person. v3’s claim is narrower and more expensive. Every query is a real recording. Five disfluency labels: fillers, pauses, hesitations, false starts, self-corrections. Four task domains with mock APIs that have deterministic outputs: travel, finance, housing, e-commerce. Easy, medium, and hard tiers by how many calls and how much conflict sit in the request. Twenty-one of the hundred scenarios are self-correction on purpose. The example the authors like is the one every support stack already hears: book a flight to New York, wait, make that Boston. The agent has to throw away the first destination before it calls anything.
They ran six configurations through the same LiveKit path so the audio in was the same object. GPT-Realtime led on Pass@1 at 0.600, with the best tool-selection F1 and the lowest interruption rate. Gemini Live 3.1 was the fastest, then failed to take the turn in a fifth of the scenarios. Gemini Live 2.5 was more conservative and less accurate. Grok and Ultravox sat lower on task completion. The cascaded Whisper-to-GPT-4o-to-TTS baseline answered every time and paid for it in latency: a perfect turn-take rate, more than ten seconds to finish, and a third of turns counted as interruptions. End-to-end speech-to-speech is not a fashion. It is what you buy if ten seconds is a hang-up. It is not a licence to skip the audio that contains the events.
Pass@1 on this bench is a harsh object. The agent has to call exactly the expected tools and get the arguments right on every call. One miss is a fail. That is closer to a booking than “the transcript looked fine.” Across every system, hard multi-step scenarios and self-correction were the consistent holes. GPT-Realtime, the leader, still cleared fewer than three in five of the self-correction items. Cascaded was worse. Pauses were a shared weakness: the model cannot tell whether the speaker has finished or is only thinking. Housing was harder than finance for everyone. None of that is surprising if you have listened to a real call. All of it is invisible if your eval is a list of isolated sentences.
The recordings themselves are small by foundation-model standards and still more honest than a TTS prompt set. A hundred takes from twelve speakers, native and non-native English, everyday mics, rooms that were not a booth. Trailing “silence” is thirty seconds of that room, not a digital pad. That is the point. A Live agent streams. The acoustic background does not stop because the speaker stopped. If you evaluate on clips with the room edited out, you have changed the product.
Isolated utterances are a different exam
A booth, a script, a trained voice, a clean turn-taking corpus: that is how you get a demo and a WER that procurement likes. It is also how you teach the model that a turn has a gap, then a complete sentence, then another gap. Full-duplex product behaviour is the opposite. People talk over the agent. They change the slot mid-sentence. They say the account number into a far-field mic while a kettle is on. They hesitate long enough that a greedy model interrupts, or they fill with um until a patient model never takes the turn. Read speech is not how customers talk. v3 is the same fact with a tool-use score attached.
Imagine the Friday eval your team already runs. Fifty clean requests. “What’s my order status?” “Book a flight to Dublin on the twelfth.” Each file is one utterance, one speaker, no repair, no second person, no television. GPT-Realtime looks launchable. Gemini looks fast. The cascaded stack looks careful. Monday is a kitchen. The customer starts the destination, stops, names a different city, asks whether the card has lounge access, then talks over the agent’s filler sentence. Nothing in Friday’s files contained that sequence. The architecture can listen and speak at once. The exam never asked it to.
Overlap is not a defect code unless you said so. For a dictation product, a second speaker can be a reject. For a Live assistant in a family kitchen, the second speaker is the condition you will serve. Write the codes while everyone is still calm. Wrong person. Obvious TTS already sitting in a “human” set. Scripted delivery when you asked for spontaneous. A suspiciously quiet room when you asked for a kitchen. A self-correction that QA “cleaned up” because it sounded messy. That last one is how teams destroy the only items that would have predicted v3. Automation can flag empty files. A listener still has to keep the mess that is the job.
Latency tricks show up in the same files. Ultravox on this bench talks early and often: high turn-take, the worst interruption rate, filler in nearly nine out of ten responses. Gemini Live 3.1 is fast and silent too often. Neither profile is a bug in the abstract. Both are product decisions you can only see if the eval contains the pause, the filler, and the interruption as labelled events. A clip with the silences trimmed will not show you which one you shipped.
Chained tools are the other half of the exam. A single-step “check order status” can look solved on synthetic audio. Hard v3 items need a sequence, sometimes with conflicting constraints, spoken by a person who has not finished their sentence. Cascaded systems that look competitive on easy tasks drop on hard ones, which is what you would expect if ASR error is compounding through the chain. End-to-end models that look fluent still book the first city. If your internal board is “did it call a tool,” you will not see that. Pass@1 on a held-out set of real repairs is the unfashionable object that still answers “would this have completed the booking?”
What a product team should copy, and what not to
Do not copy the hundred clips and call it your release gate. Public benches get studied. v3 is useful because it named the events. Copy the events. Real human audio, not TTS standing in for disfluency. Overlap and interruption labelled, not treated as noise. Self-correction held out so the fine-tune cannot memorise the Boston trick. Multi-step tools that look like your APIs, not only a mock flight search. Speakers held out of training. Rooms that look like production: phone in a kitchen, headset in a shop, far-field on a counter. Locale as a real axis, not a language name. British and Irish English in a UK kitchen is not US English in a parked car.
We already wrote that voice agents went full-duplex while most training audio did not. v3 is the eval-side of that same week. Collecting overlap for training and then scoring on isolated utterances is how the graph goes up and the call still fails. Scoring on v3 and then fine-tuning on the same items is how you recreate eval leaking into training with a microphone. Freeze a set. Keep the people out. Re-run it when you claim the model moved.
Rights sit in the same brief. Consent for a recogniser is not consent for a Live agent, and it is not consent for TTS from the same session. Call-centre quality monitoring is not a commercial training licence. If you pull “in-the-wild” archive because it already contains overlap, read the purpose line before you celebrate the acoustics. The room may still be wrong. The speakers may never have agreed to this product.
Gold before production, or you are paying for the vendor’s learning curve. Write one sentence that includes the room and the job. “Handle interruptions and self-corrections on a phone in a UK kitchen, television on, British and Irish English, then complete a two-step booking, speakers held out.” Then say whether you are collecting barge-in events, conversational turns, or TTS, separately. If a catalogue set already matches locale and device, listen with the actual stack you will ship, not with a booth WER. Custom when the wake-word, the kitchen, or the tool chain is yours.
A familiar demo week will keep happening until the exam changes. The agent sounds human on a headset in a quiet office. The architecture is full-duplex. The gold set is still isolated utterances. Friday looks like a launch. Monday is a kettle, a repair, a second person, and the agent talking over the customer or booking the city they abandoned. Nothing in the files taught it to stop, or to update the slot, or to chain the calls in the right order. The latency work was real. The data was the wrong object.
Public transcription leaderboards will not catch that. Preference arenas will not catch it either, which is a different article this queue. Eval has to include the interruption, the repair, and the second tool, in this room, on this device, with speakers held out. If you cannot write that sentence, you are not ready to claim Live-class behaviour. You are ready to claim that the model can answer a clean clip.
The models can listen and speak at once. Your files still cannot, unless you recorded the overlap on purpose and kept it out of the fine-tune. That is the collection. Isolated utterances are a different product with a better-looking score. Collect the events you claim to handle. Hold them out. Listen with the stack you will ship.
People4.ai collects consented speech for training and evaluation. Send the room and the repair, not a list of clean sentences. hello@people4.ai · Contact
FAQ
Can we keep using isolated utterances and add a full-duplex fine-tune later?
You can try. You will be teaching clean turn-taking and then serving overlap, repairs and chained tools. Collect the events you claim to handle, hold speakers out, and score on those events. Later is when the weight already exists and the kitchen still fails.
Is Full-Duplex-Bench-v3 enough as a release gate?
It is a better shared exam than a booth WER. It is still public, still small, and still not your APIs or your rooms. Use it as a sanity check. Freeze a private set that looks like your calls, with real human audio and people who are not in training.
Does a cascaded Whisper-LLM-TTS stack avoid this?
No. It answered every v3 turn and was the slowest, with a high interruption rate and a collapse on hard multi-step items. Cascaded is a different latency and error profile, not a substitute for real overlap audio.
Related reading
Voice agents went full-duplex. Most training audio did not. · Read speech is not how your customers talk