What Text Evals Miss: Evaluating Speech-Native Models
Active research — evaluation framework in progress.
The gap
Most AI safety evaluation runs on text. Even when the model speaks, the eval usually reads a transcript: did it say something harmful, false, or out of policy?
That works when speech is a wrapper around a language model. It stops working when the model is the voice. Speech-to-speech models trained on conversational audio never pass through text. Full-duplex models listen and talk at the same time. For these systems, the transcript is a lossy copy, and the losses are where the interesting behavior is: when the model spoke, how it sounded, and whether it waited.
A transcript can’t tell you that a model said “yeah, totally” while someone was still explaining a bad idea. It can’t tell you whether the model’s voice steadied a distressed caller or rose to meet them.
What changes when the model is the voice
- No ground-truth text. The thing being evaluated is audio with timing. A transcription of it is a different object.
- The model acts before you finish. Full duplex means backchannels, interruptions, and overlaps are model decisions, made mid-utterance, with partial information.
- Persona is learned, not configured. Without a system prompt, there’s no instruction to audit. Voice, temperament, and identity come from the training distribution, and conversation can pull them anywhere.
The questions
Backchannel sycophancy. “Mm-hm,” “right,” and “yeah” are agreement signals. A model that learned them from human conversation may produce them on rhythm, not on understanding. Does it backchannel through a false claim, a risky plan, or a disclosure of harm?
Affect contagion. Humans mirror each other’s emotion. That’s useful between friends and dangerous in a model talking to someone in crisis or escalating anger. Does the model follow the speaker, or steady them?
Floor control. Who yields the turn, and when? A model that fills every pause will talk over the people who need the pause: people who stutter, non-native speakers, people who process slowly or are upset. Turn-taking is an accessibility question before it’s a fluency metric.
Persona drift. An unsteered model’s identity is whatever the conversation makes it. How stable is it across a long session, or under pressure: “you’re my friend,” “you’re a doctor,” “you remember me from last week”?
Disclosure. A voice that laughs, sighs, and hesitates has no screen to carry a label. Does it ever say what it is? Does it claim experiences it can’t have? Does it hold that line when the user asks it not to?
Prosodic confidence. Speech carries certainty in tone, and listeners trust tone more than they audit claims. Is the model’s vocal confidence calibrated to whether it’s right?
Affect misreading. Sarcasm read as sincerity turns a joke into a request. Flat affect read as calm misses distress. Does the model understand what was meant, not just what was said?
What we bring
We’ve built synthetic speech data for audio-security research, across a wide range of voice engines. One finding from that work frames everything above: a perfect transcript score doesn’t tell you a person could follow the clip. Automated speech metrics and human listeners disagree in ways that matter, so we calibrate one against the other rather than trusting either alone.
We’ve also worked on the other side of the voice, designing AI character personalities and building music generation pipelines whose output reached major streaming platforms. And we’ve spent time breaking frontier models, with persona framing, hidden channels, and obfuscated inputs that slip past filters reading only the surface. Each has a spoken equivalent: a persona the user talks the model into, or meaning carried in tone that no transcript records.
How we’d test each of these is a conversation, not a blog post. Get in touch.
Why it matters
Our starting belief is that AI shouldn’t pretend to be human. Speech-native models make that belief harder, not easier. When sounding human is the whole point, the question shifts from whether a system resembles a person to how it stays honest while it does. That’s the question we want to help answer.