Voice Agents Should Act Through Interfaces, Not Just Speak

? charlie-guo’s central design claim is that builders have inherited an unnecessarily narrow model of voice interaction: a person speaks, and the system speaks back. Speech can be useful output, he says, but it is only one response channel available to software that has understood a spoken request.

The interface already has a much larger vocabulary. It can show a notification, alter application state, change the appearance of a button, highlight relevant text, or animate a cursor to show work in progress. Guo points to the ghost-cursor animation in Codex computer use as an example: the system can make its action legible by visibly moving through the interface, rather than narrating every click aloud.

A user may say what they want; the system may then carry it out and show the result in the product. In Guo’s framing, audio can become an “intelligence layer” over an existing application rather than a separate conversational shell bolted onto it.

Guo divides voice interaction into three familiar, remixable modes:

• Speech to speech: the user talks and the model talks back.
• Speech to action: the user talks and the model uses tools.
• Event to speech: the system receives an event and talks to the user.

All three have earlier analogues. The Moviefone hotline, where callers could obtain movie showtimes, can be understood as speech to speech. In-car GPS navigation is an event-to-speech system: a changing location or route condition triggers spoken guidance. The shift, Guo argues, is that models now make more interesting versions of these patterns possible—and that useful products will often combine rather than isolate them.

Speech-to-speech remains meaningful in its own right. Guo points to live practice and coaching, especially language learning, where a system can hear emphasis and emotion and give feedback on them. He also sees richer voice models improving what he calls “concierge experiences”—customer support beyond the phone-tree model—and enabling live translation with little to no latency. He imagines a future keynote being live-streamed and dubbed into multiple languages in real time.

Guo calls speech to action one of the least explored categories in voice design. He had considered titling his argument “Voice is the next capability overhang,” because he sees a large gap between what voice-capable models could do and what products currently let them do.

The immediate examples are mundane but consequential. Form filling is one. Much of online work consists of entering the same personal information into fields: names, addresses, histories, and other details required by government documents. Guo’s preferred experience is not a voice agent that reads a form back to him. It is one that lets him talk for five minutes, completes most of a document that might otherwise take an hour, and leaves him to inspect the result before submission.

He describes this as affecting a significant share of the internet’s economic activity precisely because it is boring, repeated work. The model’s job is to translate a person’s account into structured actions, not merely to provide a pleasant conversation.

Creative software is a second case. Guo says he can obtain leverage from coding systems because he can describe software in language those systems can use. But the same is not true for music production, painting, or other disciplines whose tools require a specialized operational vocabulary. A person may know the aesthetic they want without knowing Photoshop or Ableton well enough to produce it.

That produces a familiar mismatch: taste exceeds capability. Voice could narrow it by allowing someone to direct a creative process in imprecise but meaningful terms—to “vibe create,” “vibe compose,” or “vibe paint”—while a system turns that direction into work inside the tool.

The most general version is computer use. If models become able to perform on a computer the actions a person can perform, Guo asks, why should the person be limited to talking to a single app or a terminal? The longer-term interaction might be speaking to the computer as a whole, with the system selecting the appropriate tools and interfaces.

For application developers, the on-ramp is closer than that vision suggests. Modern applications already encode their capabilities as nouns and verbs: API endpoints, handlers, React hooks, workflows, settings, and records. Those existing verbs can often become model-callable tools.

This does not eliminate the need for product controls. Guo explicitly includes guardrails, safety checks, and the work of securing and productizing tool use. But his practical question for developers is straightforward: what would change if users could talk to the software they already have, rather than navigate it solely through manual input?

The current opportunity also follows from a technical shift in how voice systems are built, Guo says. The traditional architecture is a chain: speech input becomes text through transcription; the language model processes that text and may call tools; the resulting text is turned back into speech.

OpenAI’s realtime model family instead works with native audio tokens, according to Guo. Audio goes into the model and audio comes out, without transcription occurring behind the scenes. He also sees the industry moving away from a strictly turn-based interaction model toward continuous streams of incoming and outgoing audio.

The argument is not only about speed. Transcription discards features that may matter to a model’s judgment: tone, cadence, emotion, attempted interruption, and background noise. Guo invokes the familiar claim that face-to-face communication carries substantial information beyond literal words, then makes a narrower point: converting audio to text removes context that can affect how a system should understand and respond to a user.

Latency is the more quantitative case. His slide compared OpenAI voice modes in a historical 2024 comparison.

Guo presented GPT-Realtime-2 as the latest model in OpenAI’s realtime family and said that it brings reasoning to audio. Like text models, he said, it can be given more reasoning budget before responding—an answer to the familiar failure mode of a voice system replying quickly but saying something unintelligent.

That creates a trade-off. The model can think and delegate parallel tool calls, but both activities can add delay. A silent system can appear to have stalled. Guo highlights “preambles” as a way to manage that moment: developers can prompt the model to tell the user that it is about to think or call tools.

His analogy is a human travel agent. A person checking flight prices would ordinarily say they are doing so and ask for a few seconds. With preambles, an AI travel agent can similarly explain what it is checking while its tools operate in the background. The point is not simply that a voice should sound natural; it should give a user enough orientation to understand an otherwise invisible period of reasoning or action.

Guo considers event to speech more exploratory. Here, the application receives something—a message, webhook, notification, or other event—and decides whether to speak to the user.

The premise is not that every system event deserves an audible announcement. Developers already have an endless supply of logs, alerts, and notifications; showing all of them would be distracting, and speaking all of them would be worse.

Instead, Guo describes voice as the upper level of an escalation path. A system may first animate something, change visible state, or make a subtle visual cue. It may then show a more prominent pop-up. If those mechanisms do not get the user’s attention, speech can become the more interruptive channel.

Two contexts make that especially useful. The first is hands-free or screen-free interaction. Someone cooking may need help from a recipe application while their hands are occupied and their visual attention is elsewhere. The issue is not just physical inability to touch a device; it is competing attention.

The second is proactive outreach. The system may need to tell a person something when they are not actively looking at the relevant screen. The design task is to decide which events warrant interruption, rather than treating audio as the default output.

Guo’s mixed-mode examples make the point. An in-car assistant could support natural conversation through speech to speech, execute a request such as playing music through speech to action, and announce a traffic-driven reroute through event to speech. The underlying pieces already exist separately, he says, though they have not yet been combined into a seamless agent across those modes.

The same logic applies to game characters. A player might talk to a character to learn about the world, direct the character to take action that advances a story, and receive spoken reactions to changing conditions in that world. The character can converse, act, react to world events, and choose how to signal a change.

Accessibility is the broader theme Guo places beneath the three modes. He knows multiple developers who lost mobility in their hands or dexterity in their fingers during their careers and believed their programming careers were effectively over.

Large language models, coding agents, and voice agents changed that calculation for those people, in his account. They now generate orders of magnitude more code in a day or month than they previously could. Voice is not merely a convenience layer for people who can already work comfortably through conventional inputs; combined with coding agents, it can preserve a way to direct and produce software work after dexterity is lost.

That implication also sharpens the case for mixing modes. A person may need speech because their hands are unavailable, but they may still benefit from visible state changes, highlights, or other interface feedback when they can attend to a screen. The relevant design question is not whether an experience is “voice-first.” It is what combination of perception, actions, and feedback allows a person to remain effective in the situation they are in.

Guo’s proposed starting point is not, “What kind of voice agent should we build?” It is, “What is the role of voice and audio in this interaction?”

That reframing turns voice design into a set of operational decisions. What can the model perceive, and what context does it have? Which tools are available? Which actions can it execute safely and correctly? Should it communicate now or wait? When it does respond, should it speak, change the interface, or use another signal?

The resulting operating rule is narrower than simply adding a spoken interface to an app. When conversation, coaching, or translation is the value, speech-to-speech interaction makes sense. When a request maps to a capability the product can safely expose, the model can use a tool and the interface can show the resulting state. When hands or attention are unavailable, spoken interaction may be necessary. And when a system event has outgrown quieter visual signals, speech can serve as escalation.

A system that can perceive a relevant event may not need to speak about it. A system that hears a request may be better off completing a workflow than answering aloud. Voice earns its place by fitting the moment, not by becoming the mandatory reply to every spoken instruction.