Natural Voice Agents Must Balance Latency, Intelligence, and Cost

? alex-smola treats voice as an intermediate stage on the path to audiovisual agents: systems that can be spoken to, can see, and eventually appear as animated human-like avatars. That endpoint remains distant, he says, because natural interaction is far harder than merely generating intelligible speech. A conversational system can fail the moment it responds too slowly, mishandles an interruption, or delivers a technically appropriate answer in an emotionally incongruous way.

For Smola, the commercial constraint is inseparable from the interaction problem. A model can always be made smarter by making it larger, he says, but a system requiring an entire Blackwell GPU server for one streaming conversation may make an impressive demonstration without making a viable product. Boson started with a price target and worked backwards toward an architecture that could meet it.

That constraint affects every layer of a voice system. Audio must be turned into tokens, processed by a language-model-style backbone, and rendered back into sound. The more tokens the system uses per second, the more temporal detail and potential fidelity it can retain—but also the more tokens must be ingested, prefilled, and generated continuously. That drives inference cost up.

Smola contrasts spoken text, which he estimates at roughly three to five tokens per second, with audio, which can readily exceed 10 tokens per second. At 10 tokens per second, each token spans about 100 milliseconds; tokenization therefore determines the granularity at which a system can react.

The latency target comes from human perception rather than a conventional model-serving benchmark. Smola points to roughly 150 milliseconds as the time from a photon reaching the retina to cortical processing, and says auditory processing is somewhat faster. He places much of human audiovisual perception in something like a six-to-ten-hertz operating range. Boson’s goal has consequently been to make its model interruptible within about 150 milliseconds.

A designer is pushed into an uncomfortable trade-off. Higher audio-token frequency limits the parameter count that can be served economically in real time; lower token frequency permits a larger model but sacrifices temporal resolution. Buffering makes the tension sharper: a system needs enough buffer to stream reliably, while remaining able to stop or change course promptly.

Voice is not uniformly preferable to text. Smola still prefers typing for technical work involving serial numbers, URLs, or exact configuration details, where text is more precise. Voice is more useful, in his view, when someone is seeking information or advice in a social or mobile setting.

He recounts demonstrating Boson’s system at a noisy bar in Jeju, South Korea, where it handled people switching between Slovenian and Hindi. He gives significant credit to iOS noise cancellation and audio separation, and argues that a single microphone is unlikely to solve difficult multi-source environments. Proper cancellation, especially with many sources, needs microphone arrays—but, he says, that is already a solved and shipping technology.

Smola expects audio interaction to become nearly “bulletproof” within about a year. Visual presence will take longer. He expects avatars and animated robotic faces to see wider deployment in perhaps a year to a year and a half, but notes that this path requires hardware as well as software.

Once video is added, the challenge becomes real-time audiovisual streaming, with inputs arriving at different rates. Smola says current video models can produce clips on the order of 10 seconds, but an avatar needs a visually consistent feed that persists for an hour. An avatar is more tractable than unconstrained video because the system knows it is rendering an avatar, not race cars or changing backgrounds. Most frames are comparatively static and can be compressed efficiently; rapid hand movements or other large changes raise the required bitrate.

The operating point remains the same: enough quality for convincing interaction without making the service unaffordable. Biology sets the responsiveness expectation; representation learning determines how audio is encoded; systems engineering determines what can be served; and product design determines whether delays and interruptions feel tolerable. Smola’s point is that these cannot be solved in sequence.

Boson began with text and a broad focus on “AI for humans,” according to ? alex-smola. The company shifted toward audio partly because the text interface continued to feel awkward for many interactions. But the objective was not simply to train a new language model from scratch. Smola’s approach is to begin with intelligence already learned by a high-quality LLM and give it another modality.

The pattern resembles the development of vision-language models, he says: reasoning acquired from large volumes of text can be combined with visual, audio, robotic, or world-model inputs. Tokens act as the common bus through which different forms of information enter and leave the system.

The difficulty is catastrophic forgetting. If an LLM is trained only to work with audio, Smola says, it can quickly lose language and reasoning abilities acquired during text training. He compares this to a child who stops using a first language after immersion in another. The remedy is not a simple fine-tune. It requires continued language training alongside audio work, plus tasks that ground the modalities in one another.

For a multilingual model, it must learn that words such as “dog,” “xia,” “cane,” and “Hund” refer to the same concept. Audio and text need comparable alignment if the model is to reason effectively across them. Smola says this entails a full pre-training, mid-training, and post-training pipeline even where the starting point is an existing public model. A base model is a useful prior, not a finished foundation.

The degree to which Boson relies on external language models depends on the product. Smola says one model released the previous year was built on Llama, which Boson acknowledged in its model card; in other cases, the underlying model is proprietary. Whether to train atop an available model or invest more independently is both an economic and product decision, particularly where a system needs stronger performance in a specific language.

His analogy is industrial: if someone gives you free steel, do not build a steel mill; build a car. Yet adapting a language model to audio remains substantial work. Smola estimates it may take one-third to one-half as much work as the language model itself—not one-tenth.

The company’s releases distinguish among text-to-speech, automatic speech recognition, and audio understanding:

An ASR system may not know whether an utterance means “recognize speech” or “wreck a nice beach,” Smola says, because the sounds can be identical. A broader audio-understanding model can use context—a text prompt, surrounding audio, or other references—to choose between interpretations. Video could become another input alongside text and audio; robot state, self-driving-car signals, or world-model parameters could likewise enter through the same multimodal architecture.

A meaningful differentiator, Smola says, is Boson’s ability to process and train on a large audio dataset. He puts the collection at roughly 100 million hours of audio—an amount he compares with about 200 human lifetimes in a noisy environment.

The asset is not raw collection alone. Audio gathered from internet sources requires extraction, tagging, normalization, transcription, and further processing. Smola says Boson’s own data center helped make that practical because cloud storage at this scale would have been costly. The company did not rely on paying annotation companies to label 10,000-hour batches, he says; its dataset is about 10,000 times that scale.

The resulting labels are necessarily imperfect, but Smola argues that noisy data can still yield useful information at scale if errors are not systematically biased. He compares the process to repeatedly measuring an electrical outlet with a poor but unbiased voltmeter: averaging many measurements can yield a better estimate than any individual reading.

Context supplies another source of improvement. A two-person podcast, for example, offers structure that can help identify and separate speakers, producing large amounts of single-speaker audio. Models trained on the accumulated data can then create better annotations, which improve subsequent models. Smola compares the logic to using movies for face-recognition data: it may be relatively easy to identify the same actor within a single film, then use those associations to build a harder cross-film dataset with likely reliable labels.

He characterizes this as established statistical and machine-learning practice rather than a mystery technique. The difficult part is operating it at scale, with enough compute and storage to process the corpus. Smola says a nontrivial share of Boson’s resources went into data rather than training alone.

Small end-to-end audio systems have a place when responsiveness and low latency matter most, ? alex-smola says, but they are relatively limited. Making one highly intelligent becomes too expensive. The more commercially viable approach is a foreground-background architecture: the conversational system maintains the turn while understanding, reasoning, and tool use happen in parallel.

Smola compares the design to multithreaded programming. A main thread can continue the interaction while parallel work queries a database, searches the web, or calls an MCP server. The agent must decide both whether it needs that external work and how to maintain a coherent exchange until the result arrives.

Humans already make delays socially legible. If someone has to look up a date, a five- or 10-second pause can feel normal if they say they are checking. Boson’s live demo performs web search in the background, Smola says. If the result returns quickly, the system answers directly; if not, it tells the user it is looking.

That constraint shapes the performance comparisons Smola offers. He says Boson outperforms GPT, Gemini, and Grok on benchmarks including BigBench Audio and Complex Function Bench at a fraction of the cost, and says Boson is better than OpenAI’s models at one-tenth the cost. But, he adds, those comparisons use the other models with thinking turned off: a long silent reasoning interval may improve an answer yet still make a voice exchange feel unnatural.

Boson can use off-the-shelf tool infrastructure, Smola says, rather than rebuilding every service from scratch. That avoids forcing customers to adopt unfamiliar techniques. But the number of servers that can be enabled simultaneously remains limited by context-window capacity and inference cost: a large prefill and KV cache make each subsequent token more expensive. He says Boson’s next iteration is intended to support significantly more servers at once.

The model’s job is therefore more than answering questions. It has to distinguish simple requests from those requiring external work, decide when to invoke a tool, preserve the conversational flow, and keep the service inexpensive enough to run continuously.

A responsive agent should not stop every time a user makes a sound. But it should stop when the user is actually attempting to change the conversation. That distinction is easy to state and difficult to implement.

? alex-smola points to Japanese backchanneling as an example. Expressions such as “ah, sou,” “ah sou desu ne,” “ah sou ka,” and “ah wakatta” can communicate attention—roughly, “I understand,” “yes,” or “oh, I see”—without constituting an interruption. A system that treats every one of them as a request to stop will be infuriating. Conversely, if a person says they do not understand, the system should stop.

Interruptibility must therefore be scene-dependent. Smola cites two benchmarks from Boson’s Toronto team: ProAct Bench, focused on whether a model can recognize missing information and decide when to step in, and IH Bench, which measures interaction qualities including interruptibility, responsiveness, and whether audio delivery matches what is being said. The benchmark setup uses third-party audio models, he says, and is intended to make these measurements reproducible.

For builders, this expands the evaluation target beyond intelligence, transcription accuracy, and function calling. An agent can provide a correct answer while speaking over a listener, treating acknowledgement as an interruption, failing to offer needed information, or delivering an answer in the wrong vocal register. Interruption handling, delay management, proactivity, and affect have to be assessed alongside task completion.

Smola describes this as optimizing for “human happiness,” rather than only for the task-completion criteria prominent in code generation and other text-first systems. The term does not mean replacing capability with politeness. It means treating timing, responsiveness, proactivity, and affect as operational properties of the system rather than cosmetic additions.

An agent can state a fact correctly and still fail by saying it with the wrong affect. Smola invokes the famously cheerful computer in the BBC adaptation of The Hitchhiker’s Guide to the Galaxy, which announces an imminent fatal crash in a relentlessly upbeat tone. The mismatch is funny in fiction; in a voice system it is a defect.

Text-based work on emotional intelligence is useful, Smola says, but only partial. A transcript can reveal some things about whether an interruption is appropriate. It cannot fully capture tone, timing, vocal affect, or the visual layer that accompanies an avatar.

Nor is all human interaction data a good behavioral target. Smola warns against treating movies as a literal model for ordinary conduct: romantic comedies may portray stalking as successful courtship, while other genres normalize violence followed by reconciliation. He makes the same point about entertaining talk-show behavior. Systems need more careful accounts of how people should interact.

Smola says current LLMs have a decent theory of mind, which provides a starting point. He also says ethics matter: systems should not simply experiment on people. But he expects models to improve by learning how to interact with humans in appropriately designed settings.

Persistent memory is only one part of the system Smola describes. An agent can retain facts about a person—their devices, language background, preferences, and favored level of technical detail—and use them to personalize future interactions. He characterizes this as something like a glorified CRM for everyone, with a model that can review prior exchanges, reason over them, and commit useful observations into memory.

? alex-smola gives himself as an example: a system that knows he prefers equations, facts, and technical detail can adjust the next time it speaks with him. The model could also learn across interactions which behaviors help or alienate users, while adapting to regional and cultural context. Smola cites norms around eating hands in India and displaying the soles of one’s feet in Thailand as examples of behavior a globally deployed agent would need to handle with care.

Computer-only simulation offers one route to improvement before broad live deployment. Smola says Boson uses digital personas and scenarios released by Nvidia, alongside work the company has published, to train systems for abrasive, uncooperative, or adversarial users. Current models have enough understanding of people, he says, for such simulators to support some degree of recursive self-improvement.

Interaction with people would add another feedback loop. Smola describes each encounter as a potential experiment and data point from which a system could learn what works and what does not, while emphasizing that the question is how such feedback is used appropriately.

When Sam Charrington raises the problem that rare social skills may not exist in sufficient quantity in ordinary data, Smola argues that strategies can be taught through instruction and examples, then improved through observed outcomes. He mentions the “praise, critique, praise” approach to difficult feedback and management training as examples of communication techniques that can be learned.

The aspiration is not merely a polite chatbot. Smola points to the skill of a good teacher handling children who are initially uncooperative: maintaining composure, redirecting behavior, and still pursuing the task. An agent intended to complete a task should not answer hostility with hostility. It should keep its cool.