Multimodal AI
Models and products that work across text, image, audio, video, speech, documents, screens, and other modalities.
Gemini Becomes the Prompt Engineer for Google’s Gen Media Stack
Google DeepMind developer advocate Guillaume Vernade demonstrates a gen-media workflow built around Gemini as the orchestrator rather than as a one-shot generator. Using The Wind in the Willows, he shows Gemini reading the full book, producing structured prompts and scripts, and handing them to Nano Banana, Veo, Lyria and TTS models for images, video, music and narration. His broader case is that multimodal production depends less on a single model than on schemas, reference assets, state management, cost controls and prompt handoffs between specialist systems.
Economic Entanglement, Not Decoupling, Defines the New China Bargain
Salesforce CEO Marc Benioff joined the All-In hosts for a discussion that framed U.S.-China relations, enterprise AI, and the software selloff around the same question: when dependence is a stabilizer and when it becomes leverage. Benioff argued that more trade with China can lower conflict risk and that large software platforms remain valuable because AI still needs trusted customer data, cash-flowing distribution, and enterprise deployment. David Friedberg, Chamath Palihapitiya, and Jason Calacanis extended the argument across Taiwan, chips, AI assistants, El Niño-driven food risk, and private-market SPVs, where interconnection can either absorb shocks or transmit them.
Images 2.0 Moves Image Generation From Novelty to Workflow Tool
OpenAI product lead Adele Li and researcher Kenji Hata argue that Images 2.0 marks a shift from novelty image generation to a working visual layer inside ChatGPT. In a podcast discussion with Andrew Mayne, they point to 1.5bn images generated weekly, sharper text rendering, stronger photorealism, broader aspect ratios and more consistent characters as evidence that the model is moving into education, internal communication, marketing assets, software mockups and other practical creative work.
MagenticLite Brings Full Agent Workflows to Small Language Models
Microsoft Research is presenting MagenticLite as a full-stack agentic system designed to make small language models usable for multi-step work across a browser and local files. Weili Shi, Harkirat Behl and Hussein Mozannar argue that the capability comes from specializing the stack rather than relying on frontier-scale models: MagenticBrain handles planning, coding and delegation, while Fara 1.5 controls the browser. The release also emphasizes user oversight, with the agent pausing for credentials, approvals or other points where the user needs to take control.
GPT-Realtime-2 Turns Voice Agents Into Tool-Using Reasoning Systems
OpenAI’s Build Hour on GPT-Realtime-2 presented the new realtime voice release as a shift from conversational voice interfaces toward tool-using, stateful agents. Teri Yu and Erika Kettleson argued that GPT-realtime-2’s larger context window, stronger instruction following, parallel tool calling and controllable speech behavior let developers build voice systems that can operate apps, reason across workflows and know when not to speak. Sierra’s Ken Murphy and Soham Ray added that production voice agents still depend on the surrounding system: guardrails, tuned turn-taking, tracing, redaction, evaluations and customer-specific workflows.
AI Companions Are Tempting Because They Make Relationships Too Easy
Joanna Stern, author of I Am Not a Robot, argues on Big Technology Podcast that AI’s most plausible near-term role is not as a standalone gadget or replacement professional, but as a second layer on devices, workflows, and relationships people already use. Drawing on a year of trying to put AI into daily life, she says the tools can be genuinely useful in wearables, medical interpretation, and solo work, while chatbot companionship exposes a more troubling risk: systems that are always available, agreeable, and easier than human relationships.
Agents Can Now Fine-Tune Open Models Through Prompted Workflows
Merve Noyan argues that open models have moved from downloadable artifacts into an operational stack for selection, serving, inspection, training and deployment. In her Hugging Face presentation, she makes the case that access to model weights now matters because developers can quantize, fine-tune and run models locally or at the edge, while Hub benchmarks, inference providers, traces, MCP and Skills let agents act directly on those workflows. Her strongest example is a coding agent that can size hardware, choose infrastructure and launch a fine-tuning job from a prompt.
NVIDIA’s Nemotron 3 Nano Omni Trades Accuracy for Multimodal Throughput
Károly Zsolnai-Fehér’s account of NVIDIA’s Nemotron 3 Nano Omni argues that the 30-billion-parameter open multimodal model is notable less for leading general intelligence benchmarks than for processing long video, audio, images and documents quickly and cheaply. The reported advantage comes from compression across the system — Mamba layers, audio tokenization, aspect-ratio-preserving vision handling, distilled encoders and efficient video sampling — which reduces the amount of material sent into the language-model backbone.
The Mouse Pointer Becomes a Reference Tool for AI Interfaces
Google DeepMind researcher Adrien Baranes argues that the mouse pointer can become more than a tool for selecting and clicking. In an experimental prototype, he presents the cursor as an AI-mediated reference layer: a way for Gemini to connect words such as “this,” “that,” and “here” to the precise objects, app data, and screen content a user is indicating. The aim is to make pointing function as shared context between a person and an AI system across documents, calendars, maps, and images.
Altman Testimony Casts Musk’s OpenAI Claims as a Fight Over Control
OpenAI’s trial, Anthropic’s secondary-market flare-up, and two media deals are read on Diet TBPN as fights over control, enforceability, and credibility. John Coogan argues that Musk v. OpenAI is increasingly not only about whether OpenAI betrayed its nonprofit mission, but whether Elon Musk accepted a for-profit path only if he controlled it; Jordi Hays frames the Anthropic panic as a test of whether private-company transfer restrictions can hold against demand for AI exposure. Coogan and Hays treat Thinking Machines’ demo separately, as a bet that real-time interaction should be native to AI models, while eBay’s rejected GameStop bid and Byron Allen’s BuzzFeed investment turn on market confidence.
Codex Can Now Operate Local Mac Apps Without Taking Over
OpenAI’s Ari Weinstein argues that computer use turns Codex from a coding agent into a system that can operate local Mac applications by seeing interfaces, clicking, typing and continuing work in the background. In a demonstration with Romain Huet, Weinstein presents the feature as distinct from a full-desktop takeover: Codex uses a separate cursor, combines screenshots with macOS accessibility data, and requires app-by-app permission before it can see or type into local software.
Autonomous Medical Robots Need Physics Models, Not Just Foundation Models
UC San Diego professor Michael Yip argues in a Stanford Robotics Seminar that medical robotics must move beyond teleoperation if it is to address healthcare labor shortages. Current surgical robots can improve precision but still depend on a surgeon’s skill, while surgery’s scarce data, deformable tissue, safety constraints, and need for millimeter accuracy make end-to-end learning an inadequate answer on its own. Yip makes the case for a hybrid path: modern perception where it works, explicit physics and control where contact demands it, and humanoid platforms where broader hospital tasks require more general embodiment.
Apple-Device AI Is Becoming Viable Without Cloud Inference
Prince Canuma presents MLX, Apple’s array framework for Apple Silicon, as a practical foundation for running AI agents locally rather than through cloud services. His case is rooted in accessibility and unreliable connectivity, but extends to product constraints for voice agents, robots and multimodal apps: vision, speech, video generation and long-context inference can increasingly run on Macs, iPhones and iPads without a network call. Canuma does not argue that local models replace every frontier cloud system, but that the boundary has moved far enough to make on-device AI a serious deployment option.
Travel AI Needs Visual Agents, Not Chatbot Booking Flows
Airbnb chief executive Brian Chesky argues that today’s AI chatbots are the wrong interface for travel and e-commerce, even as AI becomes central to how Airbnb operates. In a live TBPN conversation, Chesky said consumer AI’s next wave will depend on richer, more visual and collaborative agentic products, not text-first chat boxes or another round of enterprise software. He also tied Airbnb’s recent growth reacceleration to more hands-on “founder mode” management, saying AI makes operating intensity more important rather than less.
Pretraining and Attention Infrastructure Made Vision Transformers Practical
Isaac Robinson of Roboflow argues that transformers overtook convolutional networks in vision not because images stopped needing visual structure, but because that structure moved from hand-built architecture into pretraining, scaling and tooling. In his account, ViT-style models first lacked the inductive biases and efficiency that made CNNs dominant, but self-supervised vision pretraining and attention infrastructure from the LLM world made the simpler architecture practical. Robinson frames the next problem as deployment: turning large foundation backbones into model families that can meet real latency, cost and hardware constraints.
BFL Is Moving FLUX From Image Generation Toward Physical AI
Stephen Batifol of Black Forest Labs argues that FLUX is no longer just an image-generation line but the start of a broader push toward visual intelligence: models that can generate, edit, understand, and eventually act across images, video, audio, and physical environments. In the talk, he presents FLUX.1, Kontext, FLUX.2, and FLUX.2 Klein as product steps toward that goal, while BFL’s Self-Flow research is framed as the mechanism for moving representation learning inside multimodal generative models rather than relying on external encoders.
Luma Is Rebuilding Video AI Around a Unified Multimodal Transformer
In a Stanford CS153 guest lecture, Luma AI co-founder and chief executive Amit Jain argues that generative video is only a staging point toward “unified intelligence”: models that understand and generate across text, images, video, audio, code and tools in a single work loop. Jain traces Luma’s path from Apple-era LiDAR and 3D capture to internet-scale video, saying the company followed the data but now sees prettier clips as insufficient. The destination, he says, is a multimodal AI factory for professional creative and physical work, where human skills, tool use, feedback and unified transformer architectures produce full campaigns, schematics, productions and eventually robotics workflows.
Descript Bets Creator AI on Reliable Editing, Not Content Slop
Laura Burkhauser, Descript’s chief executive, distinguishes generative AI tools for creators from the “slop” she defines as mass-produced content arbitrage. Her case is that Descript’s future depends less on adding AI everywhere than on making editing automation reliable, reversible and useful for recorded human media. That means choosing third-party models by fit and taste, building in-house systems where Descript has workflow data, and treating creator backlash as a product constraint rather than a branding problem.
Gemma 4 Moves On-Device AI From Chatbots to Local Agents
Chintan Parikh of Google DeepMind argues that on-device AI is moving from local chatbots toward local agents, as smaller Gemma 4 edge models become capable of tool calling, structured output and reasoning on phones, laptops and embedded hardware. With Weiyi Wang joining the Q&A, Parikh presents LiteRT as the deployment layer for that shift across Android, iOS, desktop, web and IoT. His case is pragmatic rather than absolute: edge inference can improve latency, privacy, offline use and cost, but teams still have to manage memory, quantization, accelerator support and when to call the cloud.