Full-Duplex Voice Agents Must Separate Conversation From Reasoning

Neil Zeghidour’s central distinction is between fast speech generation and a conversation that behaves like one between people. Most real-time voice models, he says, remain half duplex: the system is either listening or speaking, with the exchange divided into turns. That architecture can produce low-latency answers, but it still treats any user sound during the model’s response as an interruption.

The failure becomes obvious with backchannelingBrief acknowledgments such as “mm-hmm” or “yeah” that signal attention without claiming the conversational turn.—the “mm-hmm,” “yeah,” and similar signals people use to show attention without taking the floor. In Zeghidour’s live example, a user repeatedly tries to assure a model that they are not interrupting it, only following along. Each acknowledgment nonetheless stops or derails the model’s response. Faster latency does not solve this, he argues, because the problem is the turn-taking assumption itself.

Zeghidour says that, in a phone call with someone close to you, both speakers can be talking at once for as much as 20 percent of the interaction. Conversation includes overlap, interruption, acknowledgment, pauses in which neither party speaks, and moments in which one person begins answering before the other has fully finished a question. Those dynamics are not noise around the conversation; they are part of what makes it comfortable.

A full-duplex model must therefore do more than make a half-duplex model respond quickly. It must model two parties as independently active or inactive: both can speak simultaneously, either can be silent, and the model can continue its thought when the user signals attention rather than claiming the turn. Gradium’s presentation illustrated the contrast as clean alternating blocks for half-duplex interaction and overlapping timelines for full-duplex conversation.

In a demonstration of Kyutai’s Moshi model, the system begins answering before the speaker finishes, and the overlap continues rather than producing the brittle stop-and-start behavior shown in the backchanneling example. Zeghidour acknowledges that the early system could be irritating because it interrupted frequently. His point is not that overlap always improves judgment. It is that the model maintained a continuous conversational flow even amid noise, coughing, and concurrent speech—conditions under which a strict turn-taking system has to decide whether to stop.

The original Siri demonstration from 2011 already qualifies, in Neil Zeghidour’s broad definition, as a voice agent: a user asks for weather or the NASDAQ, and the system retrieves an answer. But its range was sharply bounded. The underlying pipeline transcribed audio, classified intent, extracted entities, tracked dialogue state, selected a skill or API, generated a rule-based or templated response, converted it to speech, and played it.

That was a closed-ended dialogue system. It could perform particular tasks in connected applications, but only those its pipeline had been designed to recognize and execute.

The next generation replaced much of that hand-engineered logic with an LLM. Zeghidour points to OpenAI’s original voice mode—not its later advanced version—as the representative system. It could sustain an open-ended interaction, such as inventing a bedtime story about a sunflower hedgehog, but had little agency: it could converse about almost anything without actually retrieving the NASDAQ or weather through an action.

The contemporary cascaded agent restores that ability. In Zeghidour’s drive-through demonstration, the system identifies a selected chicken sandwich, handles a question about menu categories in the middle of the order, adds macaroni and cheese, tracks the order, and calculates a total. Its architecture remains a cascade—streaming speech-to-text, an LLM agent, and streaming text-to-speech—but the agent layer can reason, plan, call tools, use memory or retrieval, observe tool results, and loop before replying.

That makes these systems more capable than Siri’s application-specific logic. In Zeghidour’s account, a voice agent backed by a leading text model can inherit that model’s reasoning ability and tool-call reliability. The trade-off is that the interaction remains text-mediated: transcription loses tone, emotion, and other non-linguistic features of speech, while the stages add latency.

Speech-to-speech systems collapse speech recognition, language modeling, and speech synthesis into a single model. Rather than transcribing a user’s speech into text, reasoning over text, and rendering output through a separate voice system, the model listens to audio and directly generates audio.

Neil Zeghidour identifies two immediate advantages: latency and access to information that transcripts discard. A speech-native system can respond without waiting for an entire text-based pipeline to complete, and it can use tone, emotional state, and other non-linguistic features of a speaker’s audio. He describes the responsiveness in OpenAI’s advanced voice demonstration as so strong that, in his view, making it faster would not make much sense.

But speech-native generation does not automatically mean full duplex. Zeghidour places systems such as GPT Realtime in a speech-to-speech category that is still organized around turn taking: the user speaks, then the system speaks. The system may be fast enough to feel immediately responsive, but it remains vulnerable to ordinary overlap because it must segment the interaction into listening and speaking phases.

Nor does native audio processing automatically reproduce the intelligence of a text-based agent. Zeghidour says speech-to-speech models remain “much more limited” than their cascaded counterparts. In his framing, a cascaded agent running a leading text model retains that model’s reasoning and reliable tool calling; speech-native systems exchange some of that capability for lower latency and a more natural audio interface.

Full duplex addresses the interaction constraint. It makes simultaneous activity representable rather than exceptional. The remaining question is how to do that without giving up too much of the intelligence that text models currently provide.

The technical obstacle starts with the mismatch between the sequence length of text and that of raw sound. Neil Zeghidour uses the sentence “Gradium is an AI model company based in Paris.” Spoken aloud, it is eight words and takes roughly three seconds. At a 24 kHz sampling rate, those three seconds contain 72,000 audio values.

A conventional language model predicts the next token from prior context. If it were fed raw waveforms, it would need to operate over a sequence thousands of times longer than the corresponding text. Since the self-attention cost Zeghidour describes grows with the square of sequence length, a 10,000-fold increase in length becomes a 100-million-fold increase in cost. Training an LLM directly on raw audio is therefore not viable, he argues.

The presentation’s central compression example makes the proposed solution concrete:

Neural codecs, also called audio tokenizers, supply that compression layer. An encoder transforms an audio waveform into a dense abstract representation, and a decoder reconstructs high-quality audio from the resulting tokens. Gradium’s slide describes compressing audio from 24 kHz to 12.5 Hz: the 72,000-sample example becomes 37 discrete tokens.

The codec changes the unit of modeling. Instead of predicting raw air-pressure variations, an audio LLM predicts compact audio representations much as a text model predicts text tokens. The model takes audio-token context from the user and its own previous output, then predicts the tokens it should say next.

That single sequential stream, however, still orders user tokens and system tokens one after another. It can model half-duplex dialogue, where the expected pattern is user turn followed by system turn. It does not naturally represent concurrent speech.

The full-duplex alternative is what Zeghidour calls a multistream language model. Rather than interleaving both speakers in one sequence, it places the user’s and model’s audio tokens on separate parallel streams in a multistream hierarchical transformer.

Zeghidour says Kyutai released Moshi, which it described as the first full-duplex speech-to-speech model, in 2024. He says Thinking Machines uses multistream modeling for its interaction model, and suggested that OpenAI’s planned bidirectional model would most likely use the same underlying idea. He presented the latter as an inference, not as a confirmed implementation detail.

Neil Zeghidour frames the central tension as more than an engineering-integration problem. A model has a finite number of weights—its capacity. A text model devotes that capacity to the patterns required for language understanding, reasoning, and response generation. Asking it also to understand and produce speech consumes some of the same capacity.

“Fundamentally,” he says, adding a modality to a text model carries an intelligence cost. A full-duplex speech-to-speech model may offer the highest degree of conversational naturalness, but Moshi was, by his own description, even less intelligent than speech-to-speech models that already lagged cascaded agents. A cascaded system using the best available text model, by contrast, can retain strong reasoning and reliable tool calling.

Zeghidour identifies two paths forward. The first is to scale a single end-to-end speech-native model: start with a frontier text LLM, tune it for audio interaction, post-train it, and make the resulting speech-to-speech system larger and better. The expectation is that scale, pretraining, and post-training will progressively increase intelligence while preserving naturalness and full-duplex interaction. He suspects OpenAI is pursuing this route, while noting that the company does not publish its training recipe.

The appeal is operational simplicity: one model to orchestrate, without seams between a speech interface and a reasoning backend. The difficulty, Zeghidour says, is that converting a frontier text model into a strong speech-to-speech system is expensive and complex.

The second path is to split the system. In Gradium’s MoshiRAG approach, a small, fast, potentially on-device full-duplex speech model runs the natural live conversation. Its role, Zeghidour says, is to know when it does not know and delegate the thinking, tool calling, reasoning, and agentic work to a background text model. That backend can receive asynchronous queries from hundreds or thousands of voice interfaces and return text for the spoken model to use.

Zeghidour says versions of this general architecture have also appeared in Sakana AI’s KAME and Thinking Machines’ interaction models. The split does not eliminate the tension between naturalness and intelligence; it assigns those objectives to different models rather than asking one model to bear the full cost of both.

Neil Zeghidour describes Gradium’s institutional culture as sympathetic to the “bitter lesson”: end-to-end systems, scale, and compute often outperform elaborate hand-designed structure. Yet he says the company is betting on the hybrid approach because it offers two deployment advantages: cost and control.

The cost argument is not simply that a text backend is cheaper. It is that the expensive backend need not participate in every exchange. Zeghidour calls it a loss of money to run ordinary chit-chat through a giant speech-to-speech model capable of solving differential equations. In the hybrid design, the small spoken model sustains the live conversation, while the large text model is used when the system needs deeper reasoning, tools, or other agentic capabilities. The large model can then serve requests asynchronously and in batches across hundreds or thousands of voice interfaces.

Control concerns substitutability. Developers can change the intelligence backend as text models improve rather than accepting the reasoning capability bundled with a speech-to-speech provider. They can also fine-tune a backend for particular tool calls using text data alone, without requiring audio data.

Zeghidour contrasts that flexibility with a single speech-to-speech stack. He says OpenAI’s advanced voice mode remained powered by GPT-4o until recently despite several subsequent generations of text models, because the process of updating the speech system is expensive and slow. In a split architecture, by contrast, the speech interface and the reasoning backend can evolve on different schedules.

For Gradium, the relevant standard is not a voice demonstration that sounds striking in isolation. It is a system that preserves natural, low-latency conversation while remaining economically viable and capable of using the strongest available text-model tools.