Grok Voice 3.0 just topped a composite speech-to-speech index and still trailed on the slice that looks like a support call – which is the reminder that a leaderboard is not a kitchen.
The launch landed around 14 October. Artificial Analysis put Grok Voice 3.0 at 86.9 on its Speech-to-Speech Index. GPT-Live-1 sat at 83.2. Gemini 3.8 Live sat at 82.6. Those are the numbers that travel. They sit in the launch post, the screenshot, the slide that says the stack is now first. The same board then reports τ-Voice: simulated support calls with accents, background noise and interruptions. Grok is 71.6 percent. GPT-Live-1 is 74.5 percent. The composite winner is not the winner of the call that sounds like a customer.
That gap is the market post. Not a ranking of labs. Not a shopping list of APIs. A composite index averages unlike exams. One of those exams looks like a kitchen. The others do not. If you train and evaluate on the average, you will ship the average and fail on the call.
What the index is actually averaging
Artificial Analysis does not hide the recipe. The Speech-to-Speech Index blends speech reasoning (a Big Bench Audio slice), agentic performance on τ-Voice, preference votes in a speech arena, and task-success rate. Grok Voice 3.0 is strong on the pieces that reward clean intelligence and a finished task. Speech reasoning is in the high nineties. Task success is 94.8 percent, first on the board. Time to first audio is 0.81 seconds. Those are real properties of a speech-to-speech model. They are also the properties a quiet headset in a quiet room will flatter.
τ-Voice is the other object. Accents. Background noise. Interruptions. A simulated support call, not a reasoning puzzle read into a close mic. That is closer to the product enterprise buyers keep describing: a lender, a retailer, a utility, a car, a kitchen with a kettle and a second person who is not talking to the agent. Grok trails there. GPT-Live-1 leads there. The composite still crowns Grok because the other three legs pull the average up.
A launch week always prefers the average. The average is one number. It fits a headline. It does not tell you whether the model shut up when the customer talked over the confirmation, or whether it heard the account number against a television, or whether the Irish English in the kitchen was even in the files. Full-duplex benches that score pause-and-barge-in in cleaner conditions will also flatter a stack that sounds responsive on a headset. They still are not the kitchen.
Imagine the call the index is being asked to stand in for. A customer in a UK kitchen, phone on the counter, radio on, child asking for a drink. They start the account number, stop, start again, talk over the agent, switch from a postcode to a phone number mid-sentence. The agent has to hear the repair, stop talking, keep the tool call aligned with the corrected slot, and not invent a polite confirmation over the top. Nothing in a reasoning subset teaches that. Nothing in an arena preference vote, by itself, proves the tool landed. τ-Voice is a sketch of that call. A booth word-error rate is not even a sketch.
The kitchen is a labelled event, not a vibe
Full-duplex products made this expensive to ignore. Listen and speak at once, no separate turn detector in the audio path, tools in the background while the voice keeps moving. OpenAI’s Live line and Google’s Gemini Live line have been selling that architecture for months. Grok Voice 3.0 is the same race from a third lab. The architecture assumes events in the audio: overlap, a second speaker, a kettle, a far-field phone, a repair. If those events are not in the training set, and not in the eval set, the architecture is a latency story sitting on booth files.
A booth, a script, a trained voice, a clean turn-taking corpus – that is how you get a demo that wins a composite. It is how you get a first-audio number that looks like a product. It is not how you get a support call. Read speech in a quiet room will lift reasoning scores and task-success on tidy prompts. It will not teach the model to hear a barge-in as a correction rather than as noise.
Device and room sit in the same sentence as the leaderboard. Phone in a pocket is not a headset in an office. A television is not “noise” if the product has to ignore it. A kettle is not a defect if that is where the wake-word will be said. If you collect in a booth because it is easier to QA, label the files as booth. Then do not use them as the proof that a Live-class agent works in a kitchen. Locale is still not a language name. British and Irish English in a UK kitchen is a different object from US English in a parked car. Code-switching mid-request is not an accent tag you add later. If the product claims those speakers, the audio has to contain them – including the repairs, the false starts, and the moment they talk over the agent.
Read speech, a trigger phrase, a conversational turn, and a barge-in are four jobs. Mixing them in one zip called “speech” is how a composite index looks healthy and the agent talks over the customer. We already wrote that full-duplex made the domain mismatch expensive. This week the mismatch has a scoreboard. The lab that won the average lost the call that looks like the job.
Public boards will not save a private kitchen
Public speech boards have the same contamination problem as the coding boards the industry just walked off, with a microphone. Famous items get studied. Synthetic support calls get reused. A vendor “checks quality” by fine-tuning on the eval items. A split is done by random clip instead of by speaker. Gold items that were meant to gate listeners get folded into training because they were convenient and already labelled. Friday looks like a launch. Monday is a new speaker, a new kettle, a new accent, and the product is the same as last month.
τ-Voice is still a public exam. It is a better public exam than a booth transcript. It is not your tickets, your kitchen, or your speakers. If you copy the public support-call split into a fine-tune “just to check,” you have rebuilt the board in your basement. If your vendor’s gold is also the customer demo set, you have done it with extra steps. A frozen set that looks like production is slower than a leaderboard screenshot. It is also the only number that answers whether last month’s speech fine-tune helped your assistant, in your room, on your calls.
Hold speakers out. A random split of clips from the same voices is a test of whether the stack remembers those people. Rights for wake-word, conversational training, evaluation and TTS are still separate yes or no lines. Consent for a recogniser is not consent for a clone. Call-centre quality monitoring is not a Live-agent training licence. None of that appears in a Speech-to-Speech Index cell, and none of it is optional if you ship.
What a brief has to say this quarter
The one-sentence job, including the room. “Handle interruptions on a phone in a UK kitchen, television on, British and Irish English, spontaneous, no video unless the model will see it.” Then say whether you are collecting wake-word, conversational turns, barge-in events, or TTS – separately. Rights for each. Gold before production. A listener still has to catch the cases automation will miss: wrong person, obvious TTS already in a “human” set, a suspiciously quiet room when you asked for a kitchen, scripted delivery when you asked for spontaneous.
Write reject codes while everyone is still calm. Overlap is not a defect unless you said so. A second speaker can be a reject if you asked for a single enrolled voice. It can be gold if the assistant has to ignore a child. Wrong language. Clipping. A booth when you asked for a shop. Automation can flag empty files. A person still has to apply the codes that mean “this is not the job.” If gold items do not exist before production, the first month is the vendor learning your kitchen on your budget.
If a catalogue set already matches locale and device, listen with the actual stack you will ship – the Live-class model, not a separate ASR that you will not use. Custom when the wake-word, the kitchen, or the voice-clone rights are yours. Starting custom because a leaderboard moved is how briefs balloon.
A familiar week: the agent sounds human on a headset in a quiet office. The architecture is full-duplex. The gold set is still isolated utterances from a booth. The slide shows 86.9, or 83.2, or whatever number won on Wednesday. Friday looks like a launch. Monday is a kitchen, a kettle, a second person, and the agent talking over the customer. Nothing in the files taught it to stop. The latency work was real. The composite was real. The data was the wrong object.
The models got more conversational. The audio did not magically appear. A Speech-to-Speech Index is a useful sanity check, the way a coding board is a useful sanity check. It is not a release gate for a product that lives in a room. Collect the events you claim to handle. Label the room. Hold the speakers out. Then listen with the stack you will ship, not with a booth WER and not with an average of four unlike exams. If the gold set is still isolated utterances, you are measuring a product you are not shipping. If the only number you can show is the public index, you are measuring someone else’s exam.
People4.ai collects consented speech for training and evaluation. Send the room, not a screenshot of a leaderboard. hello@people4.ai · Contact
FAQ
Does topping a speech-to-speech index mean the model will handle our support calls?
No. A composite averages unlike exams. Grok Voice 3.0 led the index this month and still trailed on τ-Voice, the slice that looks like accents, noise and interruptions. Your calls are a private exam. The public average is a press cycle.
Can we fine-tune a Live-class model on booth audio and fix kitchens in production?
You can try. You will be teaching clean turn-taking and then serving overlap. Collect the events you claim to handle – barge-in, a second speaker, the actual device – or stop claiming them.
Is τ-Voice enough eval if we do not want to build a private set?
It is a better public sketch than a booth transcript. It is still public. Famous items get studied. Hold your own speakers out, freeze a set that looks like your room, and re-run it when you claim the model moved.
Related reading
Voice agents went full-duplex. Most training audio did not. · OpenAI just said the coding benchmark was contaminated