Multimodal AI
Models and products that work across text, image, audio, video, speech, documents, screens, and other modalities.
Codex Turns Everyday Rituals Into Deliberately Excessive Prototypes
At Hahahacks, a gathering presented as “Experiments” by OpenAI and TIAT, participants used Codex Pro to turn deliberately absurd premises into working prototypes. The projects formalized minor social rituals with computer vision, interfaces and exaggerated physical responses: a shoe-removal rule becomes an entry checkpoint, routine mail gets a cartoon announcement, and an alleged lie triggers water to the face. The event’s argument is that Codex can move quickly from a strange prompt to a tangible app, even when the point is not efficiency but theatrical excess.
Reliable Information Is a Prerequisite for Self-Government
Vivian Schiller, Aspen Digital’s executive director, argues that the program’s central concern is whether people can access the reliable, independent information needed for self-government—not simply whether journalism institutions survive. She traces Aspen Digital’s work from its communications-policy roots through cybersecurity, local news and AI, contending that technology now shapes public understanding across nearly every policy field. Her chief AI concern is not a single viral deepfake but the broader “liar’s dividend”: a flood of fabricated or contested material that leaves people unsure what is real.
Robotics Still Lacks Its ChatGPT Moment
Google DeepMind’s head of robotics, Carolina Parada, says recent advances in language and vision models have moved robots beyond treating their surroundings as meaningless coordinates toward understanding instructions, environments and tasks. But she argues robotics has not yet reached its “ChatGPT moment”: a system that can reliably respond to a person’s request in an unfamiliar setting. Humanoids may help deploy such intelligence in spaces built for people, she says, though many tasks will call for other robot forms.
FLUX 3 Launches Multimodal Video Generation With Native Audio
Black Forest Labs presents FLUX 3 as its first video model, trained jointly on images, video and audio to generate up to 20-second clips with dialogue, sound effects and ambience created alongside the frames. The source argues that its Self-Flow training approach improves semantic understanding without a separate vision model, underpinning claimed gains in motion, faces, hands and in-scene text. It positions video continuation and low-cost Draft mode as the most practical workflow features, while noting that the preview release lacks editing and reference support.
Gemma 4 Makes Perception Part of the Main Transformer
Károly Zsolnai-Fehér argues that DeepMind’s Gemma 4 offers a practical design pattern for smaller multimodal models: its 12-billion-parameter version feeds raw image patches and audio segments directly into the main transformer rather than relying on separate perception encoders. He says the encoder-free architecture makes a single model responsible for seeing, hearing and reasoning, cutting specialist parameter overhead while preserving multimodal capability. Because DeepMind published the design, he argues, other open-model developers can adopt it rather than pursuing scale alone.
Qwen 3.8 Max Could Pressure Closed AI Providers on Price and Openness
Qwen 3.8 Max could put pressure on OpenAI and Anthropic not by leading every benchmark, but by combining frontier-style multimodal agents with lower claimed API prices and planned open weights, argues Two Minute Papers’ Károly Zsolnai-Fehér. He presents the model as capable of carrying out and revising work over days, while acknowledging that its displayed software-engineering scores trail leading closed rivals. The more consequential prospect, he says, is whether smaller Qwen releases can bring comparable capability to local users.
A Red-Line Guide Turns Aerial Images Into Virtual Drone Flights
ElevenLabs presents a workflow for generating drone-style video from a single AI-made aerial image, using a red line drawn over the image as a virtual camera route. The guide line tells the video model where to move through the scene, while the prompt specifies motion, pacing and visual action; the tutorial positions Gemini Omni Flash as the faster, cheaper option and Seedance 2.0 for longer, higher-resolution shots with native audio.
Apple Prepares Siri AI Smart-Home Hub for Fall
Apple is preparing a smart-home hub with a roughly seven-inch display that would put Siri AI at the centre of voice control, video calls and connected-device management, Bloomberg’s Dana Wollman reports. Wollman says Apple delayed its entry into the category while its software teams caught up on AI, but now expects the company to unveil new home devices this fall. The reported hub would target the shared household-screen role established by Amazon Echo and Google Nest products.
AI Agents Need Bounded Loops, Observable Goals, and Approval Boundaries
OpenAI developer-experience lead Jason Liu argues that Codex is most useful when treated as a set of durable, bounded workstreams rather than a succession of disposable chats. His model starts with a pinned thread that retains context and reports on a defined condition, then adds persistent goals only where completion can be verified, and durable memory, skills and broader computer control only when repeated work warrants them. The governing constraint is explicit: give the system enough context to prepare useful work, but keep action and approval boundaries clear.
Robot-Ready Blender Scenes Require Physics, Sensors, and Semantic Labels
NVIDIA argues that 3D scenes built for visual realism must also be given physical, sensor and semantic properties before they can train robots. Its Omniverse libraries, now part of the NVIDIA Agent Toolkit, are designed to bring that validation work into Blender: in the company’s demonstration, an agent adds rigid-body physics, LiDAR-style sensor output and object labels to a classroom before it is handed to Isaac Sim. The aim is to let creators prepare robot-ready environments without leaving their existing authoring tools.
NVIDIA Cosmos Uses Compute to Generate Training Data for Physical AI
NVIDIA argues that physical-AI developers cannot collect enough real-world data to cover an “infinite and unpredictable” environment, and that compute must therefore generate the missing training material. Its Cosmos model combines an autoregressive reasoner with a diffusion generator to process and produce pixels, actions, sound and language, generating physics-aware scenarios and action-conditioned futures. NVIDIA says post-training adapts the model into a world action model for particular vehicles, robots and other physical systems.
World Models Aim to Replace Robotic Trial and Error
Ankit Gupta and François Chaubard argue that AI needs world models—systems that predict how an environment will change after an action—to escape the poor sample efficiency of trial-and-error learning. They contend that passive video could provide broad knowledge of physical change, then be paired with smaller action-labeled datasets to train robots in simulated rollouts. But Chaubard says the approach still faces hard limits in large action spaces, rare safety-critical events, real-time planning, and adaptation when physical conditions differ from the model’s predictions.
Controlled References Turn One Photo Into Broadcast-Style Football Footage
ElevenLabs’ Alec Wilcock argues that convincing AI football footage depends on production control rather than a single successful video generation. His workflow uses a face image and multi-angle character sheet to hold identity and kit constant, plans keyframes around a three-shot edit before animating them, and fixes visual drift at the image stage. Video, stadium sound and commentary remain separate layers so creators can trim and revise the sequence without rebuilding it.
Stripe’s $53 Billion PayPal Bid Hinges on Operational Turnaround
John Coogan argues that Stripe’s $53 billion offer for PayPal is a wager that its consumer accounts, bank-linked relationships, checkout presence and cash flow can be worth far more under stronger management. He and Jordi Hays say the transaction depends less on identifying those assets than on whether Stripe, alongside Advent International, can restructure and integrate a 25,000-person legacy fintech without eroding them. The show applies a similar test to OpenAI’s reported AI speaker and Chip Motors’ autonomous neighborhood EV: appealing concepts still have to clear the harder work of building, operating and delivering them.
Codex Expands From Code Generation to Software Work Coordination
OpenAI’s Romain Huet argues that Codex is evolving from a code-generation tool into a workspace for delegating and coordinating software work across planning, implementation, testing, deployment and pull-request review. The update centers on GPT-5.6 Sol and its higher-budget Ultra mode, which can split broad goals into parallel tasks, alongside visual browser and computer controls, inline edits, Sites deployment, and task management across desktop and mobile. Huet also notes that Ultra’s larger reasoning budget consumes token limits faster.
Seedream 5.0 Pro Adds Sketch-Guided AI Image Editing
ElevenLabs positions Seedream 5.0 Pro as an image-generation model for work that needs revision after the first render, rather than another prompt-only generator. The company says users can direct edits with sketches and annotations, generate text-heavy layouts in 14 languages, and—after launch—separate a composition into transparent, reusable layers. The model is available at 2K resolution, while 4K output and layer separation are not yet available.
Gemini Omni Flash Turns Prompted Briefs Into Full-Context Video Edits
Alec Wilcock presents Google’s Gemini Omni Flash in ElevenCreative as a prompt-driven video editor that can alter footage while preserving the original shot’s motion, framing and subject consistency. His central argument is that the model works best when prompts read like edit briefs: specify the change being requested, then spell out what must not change. Across examples including restyling, relighting, object replacement, subtitles, titles and full Reel edits, Wilcock treats precision and constraints as the difference between a usable edit and a generic regeneration.
GPT-Live Turns Voice Requests Into Task-Specific Interface Widgets
OpenAI’s GPT-Live demo presents voice as a control layer for changing on-screen information, not just as a spoken answer. In a Mexico City planning thread, the model turns conversational requests into a weather card, a World Cup group schedule and then a map of sports bars, while keeping the spoken response shorter than the visual detail. The source argues that the important behavior is continuity: GPT-Live carries context across tasks and reshapes the interface around the user’s next decision.
GPT-Live Applies Conversational Context to Image-Based Outfit Advice
OpenAI’s GPT-Live image interaction demo presents the model as doing more than identifying what a user is wearing. In a visual chat about outfits for meeting a girlfriend’s parents, the assistant evaluates two photos against the same social constraint, first warning that a patterned shirt may read “a tiny bit loud” and then treating a muted overshirt as more approachable. The point of the example is multimodal continuity: the model’s second answer depends on the prior occasion, not just the new image.
GPT Image 2 And Aleph 2.0 Replace Video Objects From One Still
ElevenLabs’ tutorial argues that object replacement in video can be reduced to a frame-based workflow: use GPT Image 2 to alter a clean still from the source clip, then give that still to Aleph 2.0 as the target look for the full video. The demonstration replaces an apple with a flaming potato while preserving the rest of the shot, and extends the same method to background swaps and green-screen-style edits. The source’s practical caveat is that results depend on a clear reference frame and crisp footage with limited motion blur.
Creator Businesses Are Sorting by Cost, Distribution, and Control
John Coogan and Jordi Hays argue on Diet TBPN that several hyped technology markets are entering a more exacting phase in which distribution, margins and access matter more than slogans. Their discussion frames the creator economy as a sorting of business models, Meta’s smartglasses as a consumer hardware category gaining traction despite weak investor credit, and OpenAI’s limited GPT-5.6 rollout as evidence that frontier AI is now constrained by security policy, infrastructure and control over who gets to use it.
Perplexity Brings Computer Agent Workflows Into the Mac Desktop
Perplexity Academy presents Personal Computer as a Mac-based extension of Perplexity Computer, designed to let the system act on the files, apps, folders, browser tabs, and desktop context where users’ work already sits. The tutorial argues that the product’s significance is local access: users can start tasks from the Mac app, a floating desktop bar, voice mode, remote control, or Comet, while heavier processing remains on Perplexity’s servers. Its examples frame the Mac not as an endpoint for AI output, but as an operating surface for organizing folders, updating files, reviewing presentations, generating briefs, and running recurring background work.
Flows Agent Turns Creative Prompts Into Editable Multimodal Media Workflows
Flows Agent is presented as a conversational workspace for turning a creative prompt into an editable multimodal media workflow. The source argues that its value is not just generating a single video, image, or audio asset, but assembling a structured flow of voice, image, video, and sound nodes that users can revise through natural-language prompts, replace with uploaded assets, and later customize or export.
Flows Agent Turns Creative Briefs Into Editable AI Production Pipelines
ElevenLabs presents Flows Agent as a conversational assistant for building and revising node-based creative workflows inside ElevenCreative Flows. The company’s case is that a user can describe an ad or other asset in natural language, have the agent assemble the models, prompts, nodes, and connections, then keep the resulting pipeline visible for edits, approvals, and reuse. The demo emphasizes cost controls for credit-heavy generation, node-level revisions through chat, and templates that turn a completed flow into a repeatable production system.