Human preference Elo and task-success scores on voice agents are different objects, so a Speech Agent Arena win is not proof the model completed the job – and the same batch should not be used as both preference data and the frozen eval.
October’s voice leaderboards made the split visible without a lab having to admit it. Artificial Analysis’s Speech Agent Arena ranks speech-to-speech models on blind, live conversations: booking a dental appointment, ordering takeout, asking about opening hours. People talk to two hidden systems and vote which they preferred. That vote becomes an Elo. Separately, on agentic scenarios, the same programme scores whether the model actually completed the action through the correct final tool call. Tau Voice, the agentic board that sits next to those arena numbers, asks a harder version of the same question: can the spoken agent run the multi-step job. Gemini can sit at the top of preference and well down the agent-task table. That is not a ranking error. It is two exams.
Gemini 3.1 Flash Live Minimal has been leading Speech Agent Arena preference, north of 1100 Elo in the October snapshot, while completing about three in four eligible agentic tasks on that arena and scoring in the mid-twenties on Tau Voice. Gemini 3.8 Live sits near the top of preference with a high arena task-success rate and still only about 30 percent on Tau Voice. The Extended Thinking variant reverses the picture: much stronger on Tau Voice, clearly weaker on preference Elo, because a model that reasons longer is not always the one people like talking to. GPT-Live Astra leads or sits near the top of Tau Voice while trailing Gemini on the arena vote. Grok can win arena task success and still not win the conversation people would pick. If your slide has one number, you have already chosen which product you are pretending to have.
This is the market. Live agents are being bought on feel and shipped on tools. Preference data makes the feel. Task-success eval measures the booking. Mixing them in one zip called “voice eval” is how a launch sounds great and the takeout never arrives.
What each board is actually scoring
Speech Agent Arena preference is a pairwise human vote after live calls. The methodology notes that highly preferred models tended to respond quickly, sound more natural, and produce fewer artefacts. That is a real product axis. People hang up on a voice they dislike. It is not “did the appointment get booked.” Arena task success, on the agentic subset, is closer: share of eligible conversations where the required final tool call happened. The same write-up already said the quiet part. Some conversations sound as if the action completed when the tool call did not. A preferred model can fail roughly one eligible task in four and still win the vote. That is not a Gemini problem. It is what happens when fluency and completion are allowed to disagree.
Tau Voice is the harsher agentic object. It is a spoken, tool-using exam, not a vibe check. Extended Thinking helping Tau Voice while hurting Elo is the same disagreement with a reasoning knob. Product teams who only watch preference will ship the model that chats. Product teams who only watch Tau Voice will ship the model that pauses, thinks, and loses the call on feel. You need both numbers. You need them on different items.
Public boards have a second problem this year, which we already covered when labs walked off contaminated leaderboards. Anything famous enough to put on a launch blog gets studied. Arena conversations and Tau Voice items will leak into training mixes the same way SWE-bench tasks did. A Gemini win on a public Elo is not proof your dental-booking agent got better on your tools, in your locale, with your speakers. Private eval is becoming the product for the same reason. Private eval that looks like production is the unfashionable object that still answers “did this release help?” Preference Elo is a press cycle. Task success on a frozen, held-out set is a release gate. They are not interchangeable.
Here is the in-house version, without a famous board. A vendor collects two-thousand preference pairs on “which voice assistant would you rather talk to.” Raters hear two completions. They pick the warmer one. The same pairs are later reused as the quarterly metric for “did the agent complete the task.” Completions that sounded finished win. Tools that never fired are invisible. The graph goes up. Support tickets do not. Someone then fine-tunes on the same batch “just to check quality.” Friday’s demo is a lovely call. Monday the booking is still wrong, and you can no longer use those items as an exam because they sat in the weight.
That pattern is older than Speech Agent Arena. Preference data teaches a model which answers to prefer among behaviours it can already produce. Task-success eval measures whether a change actually completed a job on data the model was not trained on. Human eval in the buying-guide sense is the third object: held-out prompts, a rubric, scores, not used as training. Most teams need more than one. Almost no team should use the same files for all three.
Do not use the same batch as both
Write the objects on separate lines before anyone records.
Preference for a voice agent is a vote on a live or recorded conversation: which system would you rather keep talking to, on named axes. Natural, fast enough, not sycophantic, not full of filler, not interrupting. Raters need gold items first. Discard is a valid label when both sides failed the job. Forced choice on two failed bookings teaches the model which flavour of failed booking to prefer.
Task-success eval is a rubric on whether the tools fired correctly and the slots ended in the right state. The speaker changed the restaurant. Did the order match the second choice. The account number was repaired mid-sentence. Did the agent use the repaired number. Overlap, locale, device, and the actual API schema belong in that rubric. A thumbs-up on “helpfulness” will not see any of it.
Training preference pairs and a frozen task-success set must be different items, and usually different people. Speakers leak. Style leaks. If the same contractors wrote the SFT, voted the preferences, and scored the quarterly board, you have contamination even when the transcripts look different. If you split by random clip instead of by speaker, the model has already heard those voices. Eval leaking into training does not become safe because the modality is audio.
Rights are separate yes or no lines as well. Permission to collect a preference vote is not permission to train, and it is not permission to keep a frozen exam and re-run it for years. Consent for a recogniser is not consent for a Live agent. TTS from the same session is another product. If the SOW says “AI use,” you do not have those lines.
A familiar week in a voice lab: product wants the Gemini-shaped feel because the arena Elo is the screenshot in the board pack. Engineering wants the Astra-shaped or Extended-Thinking-shaped agent because Tau Voice is the only number that looks like a booking. Someone proposes one collection to settle it. “Get humans to talk to the models and tell us which is better, and we will also score whether the task completed.” That sentence is two datasets. If you pay for one session and use it twice, you will fit the vote and spoil the exam, or freeze the exam and starve the preference model of the very conversations you needed. Budget for both. Label them. Hold the eval people out.
Synthetic stand-ins make the cheat easier. TTS reading a scripted repair is not a person changing their mind in a kitchen. A model-as-judge on “did it sound like it booked” is the student sitting the exam. Use it for triage. For the number you show a customer or a regulator, use people, or say clearly that you did not. Synthetic audio can fill a pattern you already understand. It is a weak frozen eval of a product that will hear real customers.
Locale and room still sit under both objects. A preference vote in US English on a headset is not a preference vote in Irish English on a far-field kitchen mic. Task success on a mock takeout API is not task success on your insurance schema with till noise. If the product claims those rooms, both the preference set and the task-success set have to contain them. Copying October’s public ranking into a slide does not transport the acoustics.
What to put in the brief this month
Two sentences, not one zip.
“Preference: live or recorded pairwise votes on named conversational axes, speakers held out of training, rights to train on the votes.”
“Task success: frozen held-out scenarios that look like production, including repairs, overlap and our tools, different people, rights to keep and re-score, not to train.”
If you cannot write the second, you are not ready to claim the agent works. You are ready to claim people liked talking to it. If you cannot write the first, you may ship a correct booking that nobody will stay on the line for. Both can be true. Gemini just showed they can point in different directions on the same week’s boards.
Gold before production. Rubric before raters. A rule for disagreement. A rule for “both failed.” Split by speaker. Do not recycle the preference batch as the board metric. Do not fine-tune on the task-success items to “check quality.” Public Elo can stay as a sanity check. Tau Voice can stay as a sanity check. Neither is your dental office, your kitchen, or your schema.
The models got better at sounding like a call. The booking did not automatically follow. Preference and task success are different purchases. Collect them as different objects, or stop treating the leaderboard screenshot as a release.
People4.ai will collect preference and eval as separate packs when you ask for them that way. hello@people4.ai · Contact
FAQ
Can we use Speech Agent Arena Elo as the release gate?
As a sanity check, yes. As proof your agent completed your tasks, no. Preference and task success already disagree on that public board, and the items are public enough to get studied. Freeze a private set that looks like your calls.
If people preferred the model, did it not complete the task?
Often it sounded as if it did. Arena write-ups already note conversations that feel finished when the required tool call did not happen. Score the tool state separately, on different items, with speakers held out.
Can one vendor session give us both preference pairs and the quarterly metric?
Not if you want a held-out exam. You can collect both kinds of judgement in one operation if the files, people and rights are split on purpose. Using the same batch as training data and as the board metric is how the graph lies.
Related reading
Private eval is becoming the product · OpenAI just said the coding benchmark was contaminated