A word-error rate on a booth clip is the wrong exam for a full-duplex voice agent. Interruption, overlap and tool use while the audio continues do not show up in a transcript.
OpenAI has now written that down as an evaluation guide for GPT-Live. A full-duplex agent has to sustain a conversation and complete a task. It can listen, speak, take a correction, and delegate work while the audio is still running. Evaluating only the transcript, or only the final text, misses the failures that actually kill the product: the agent talked over the customer, missed the repair, confirmed a booking before the tool landed, or kept speaking into a silence that was a thought, not a turn. A natural-sounding confirmation does not prove the backend did the right thing. A clean transcript does not prove the agent shut up.
That is the market post. Not a new definition of WER. The frontier voice stacks spent a year making listen-and-speak-at-once the product. The eval habit is still a booth clip and a word-error rate, or a chat log of what the model would have said if this had been text. Full-Duplex-Bench-v3, on arXiv, is the research version of the same complaint. The set is real human audio, annotated for fillers, pauses, hesitations, false starts and self-corrections, paired with chained tool use. Prior voice-agent boards either skipped tools, or used synthetic speech, or both. WER on a quiet read is not in that sentence, because it was never the exam.
What the guide is actually asking you to score
OpenAI splits the job into two axes that can fail independently. Audio and interaction: did the frontend hear and speak, on time, without talking over the caller, including backchannels and a spoken result that matches the backend. Reasoning and task: did the system understand the request, delegate the right work, execute the right tools, and leave the application in the right state. Natural speech cannot rescue a wrong booking. A correct booking cannot rescue an agent that barges in for four seconds of overlap. A transcript score collapses those axes into one string of words.
The harness comes in three modes. Crawl uses synthetic audio and a controlled single-turn request, so you can see whether the tools and the state machine work when the acoustics are not the variable. Walk replays a saved recording, so the same request has to survive a real voice, a real room, a real microphone. Run puts an independent simulated caller into a continuous multi-turn conversation, so interruption, overlap and a change of mind happen on a clock that does not pause between turns. The same assistant, the same tools, the same application. Different input. If you only run Crawl, you have a text agent with a TTS wrapper. If you only score the transcript of Run, you have thrown away the reason Run exists.
The running example in the guide is a restaurant reservation. The caller says they want a table for Maya on 7 August – sorry, 6 August – at 7pm for two. A pass is not “the transcript looks like a booking.” A pass is one reservation, on the sixth, after the correction, confirmed only once the tool succeeds. Booking the seventh, booking twice, or saying yes before the API returns, is a fail even if every word was recognised. That is the whole argument in one clip. ASR can be perfect. The product can still be wrong. The product can also be right on state and still be unusable if the agent talks over the “sorry.”
Full-duplex makes the clock part of the exam. User speech, assistant speech, interruptions, delegated reasoning and tool execution can overlap. The harness has to sit on a continuous timeline. It cannot pause between turns the way a chat eval pauses. Silence during delegation is a metric, not a vibe. Interruption rate is a metric. Time to first audio is not time to a finished task. Teams that only report WER are reporting a third object again: whether a recogniser heard a booth. GPT-Live is not a recogniser. It is an agent that speaks while it works.
Real disfluency is not a defect code unless you said so
Full-Duplex-Bench-v3 is useful because it refuses the tidy clip. Twelve speakers, native and non-native English, everyday mics, rooms that are not studios. Trailing silence is thirty seconds of the speaker’s actual ambient, not digital quiet pasted on the end. The hard items are self-corrections and multi-step tool chains: discard the first city, keep the second, do not hallucinate a third API call. The paper’s own results are the usual frontier mess. One stack leads on task completion and talks over people less. Another is faster and worse at taking the turn. A cascaded Whisper-then-LLM-then-TTS pipeline can look polite and still be too slow to feel like a call. Self-correction is where they all sag.
That is not a leaderboard you should copy into a launch blog. It is a description of the object you should be collecting. Fillers, false starts, mid-utterance changes of slot, a pause that is not the end of a turn. If your eval set is isolated utterances from a booth, you are measuring a product you are not shipping. If your training set is the same booth, you are teaching the product you are measuring. We wrote the domain version when the files were still called “speech data”: read speech is not how customers talk. The news this month is that the labs have started scoring the events the architecture claims to handle. Your gold set has to contain those events, labelled, with speakers held out.
QA for a booth set looks for clipping, wrong language, a quiet room. QA for a full-duplex eval has to keep the mess that is the job. A second speaker can be a reject if you asked for a single enrolled voice. It can be gold if the assistant has to ignore a child. A self-correction is not a failed take. It is the item. Write the codes while everyone is still calm. Scripted delivery when you asked for spontaneous. Obvious TTS already sitting in a “human” set. A suspiciously quiet room when you asked for a kitchen. Automation can flag empty files. A listener still has to catch the rest. If gold items do not exist before anyone scores, the first week of eval is the vendor learning your kitchen on your budget.
Hold speakers out of both training and eval. A random split of clips from the same voices is a test of whether the stack remembers those people. Public transcription leaderboards will not catch that. They have the same contamination problem as SWE-bench, with a microphone. If a vendor “checks quality” by fine-tuning on the eval items, you no longer have eval. You have more training. If the same three contractors wrote the prompts and the rubric in the same week, you have style leakage even when the WAVs look different.
What to freeze instead of a WER cell
A frozen pack that looks like production. Different items from training. Different people. A rubric that includes interruption, overlap, tool state and the spoken confirmation – not only the words in a transcript. A disagreement rule. Rights that say you may keep the set, show scores, and not train on it. If the SOW only says “AI use,” you do not have that line. Training, evaluation and shipping are separate yes or no answers. Eval you cannot keep is a one-off demo, not a product metric.
Write the failure modes into the rubric before you collect. JSON schema skipped. Policy invented. Wake-word missed with the television on. Booking confirmed before the tool returned. Agent talked over the repair. Date stayed on the first number after the caller said sorry. Then freeze. Size is second. A small set you can read in an afternoon, and will actually re-run, beats a vast board nobody opens. Voice models are nondeterministic. The labs now say to run the same scenario more than once and report a range. Copy that instinct. Do not copy a public board’s items into the fine-tune “just to check.”
Crawl can stay as a functional baseline. It is synthetic on purpose. Label it as synthetic. It is not a human baseline, and it is not the kitchen. Walk is where your recordings earn their keep: the actual device, the actual room, the actual accent. Run is where you find out whether the agent is a call or a chat window with speakers. Human review still sits on top for the qualities a judge model will flatter – pace, whether the interruption felt rude, whether the accent was handled or steamrolled. A model-as-judge can triage. It is still the student if it was trained on the same mix. Keep a human sample for the number you show a customer or a regulator.
A familiar Friday: the transcript WER is fine. The demo on a headset in a quiet office sounds like a person. Someone pastes a Speech-to-Speech Index cell into the slide. Monday is a kitchen, a kettle, a corrected date, and a booking on the wrong night because the agent never treated “sorry” as state. Nothing in the exam taught it to. The architecture was full-duplex. The eval was a booth. The tool log, if anyone opened it, was the only honest sheet in the building.
If you need a set that looks like your product and is not in the crawl, ask for it as a separate pack. The one-sentence job, including the room. Failure modes in the rubric before anyone scores. Speakers held out. Rights to keep the set and not train on it. A freeze date. Re-run when you claim the model moved. Do not wait for a famous board about your kitchen. Your tickets, your speakers, your repairs are the only exam that cannot already sit in a public clip – unless you publish them, or you train on them, or you let a transcript stand in for the call. WER can stay on the wall as a sanity check for the recogniser you may not even be shipping. It cannot be the claim that a Live-class agent works. Score the interruption. Score the tool. Score the room. If you cannot write those three lines, you are not ready to claim full-duplex. You are ready to claim a transcript.
People4.ai will collect a frozen voice eval that looks like the call when you name the call. hello@people4.ai · Contact
FAQ
Can we keep using WER as the main voice-agent metric?
Use it as a sanity check on recognition if you still have a recogniser. It will not tell you whether the agent handled a barge-in, kept the corrected date, or confirmed after the tool succeeded. Full-duplex eval has to follow the request from speech to application state.
Is synthetic Crawl audio enough to ship?
Crawl tells you the tools and the state machine work in clean conditions. Walk and a held-out human set tell you whether that survives a real room. Shipping on Crawl alone is shipping a text agent with a voice.
Are public full-duplex benches a substitute for our own eval?
They are a better public sketch than a booth transcript. They are still public. Famous items get studied. Freeze your own speakers, your own tools and your own kitchen, and do not train on that pack.
Related reading
Private eval is becoming the product · Read speech is not how your customers talk