NVIDIA Cosmos Uses Compute to Generate Training Data for Physical AI
NVIDIA argues that physical-AI developers cannot collect enough real-world data to cover an “infinite and unpredictable” environment, and that compute must therefore generate the missing training material. Its Cosmos model combines an autoregressive reasoner with a diffusion generator to process and produce pixels, actions, sound and language, generating physics-aware scenarios and action-conditioned futures. NVIDIA says post-training adapts the model into a world action model for particular vehicles, robots and other physical systems.
NVIDIA’s premise: physical AI needs data that cannot be collected at real-world scale
NVIDIA describes the physical world as “infinite and unpredictable,” while arguing that physical-AI development requires data that real-world collection cannot scale. Its proposed answer is to use compute to produce a different kind of training material: simulated scenes, inferred actions, and plausible next states.
“For physical AI, compute is data,” NVIDIA says. The claim is not that computation merely processes existing sensor records. It is that a model can generate physical-world data where recorded experience is limited.
A driving example makes the intended use concrete. In a residential-street scene, the displayed reasoning identifies a ball rolling into the road as a possible hazard because “a child or pet” may follow it. The associated meta-action is to decelerate and stop if the ball or pedestrians pose a risk. The example centers on anticipating a consequential possibility before the possible pedestrian or animal is visible.
For physical AI, compute is data.
NVIDIA positions Cosmos as an open foundation model for developers building such systems.
The architecture separates a reasoner from a generator
Cosmos is described as an open “omnimodel” built on a new mixture-of-transformers architecture. The architecture shown places text, image, video, audio, and action on both the input and output sides of the model.
NVIDIA divides the system into an Autoregressive Reasoner and a Diffusion Generator. Pixels, actions, sound, and language feed the autoregressive transformer, which NVIDIA says reasons, plans, and instructs the diffusion transformer. The diffusion component generates what comes next.
The mirrored modality labels present Cosmos as a model that takes in and produces multiple forms of physical-world context, from language and imagery to video, audio, and action.
Developers can post-train Cosmos across embodiments and use cases. NVIDIA depicts four roles around the same core model: World Reasoner, Data Generator, Simulator, and Policy Model. Those labels correspond to the broader workflow the presentation describes: interpreting scenes, producing synthetic material, generating action-conditioned futures, and using those futures for policy training and evaluation.
Observation, synthetic generation, and simulation serve different development tasks
As a vision-language model, Cosmos is presented as a system that watches the physical world, describes scenes, and flags what matters. A drone view of an urban intersection illustrates that role. Prompted to generate a report for a traffic engineer, the displayed output identifies multiple traffic lanes and crosswalks, periodic bottlenecks, and pedestrian activity concentrated at marked crossings and green-painted paths.
The report recommends adjusting signal timing to reduce intersection waits, especially for turning lanes during peak periods; extending crossing times or adding signals at high-traffic crosswalks; and reconsidering turning-lane placement to reduce conflicts with straight-moving traffic. Those recommendations are generated from the timelapse shown on screen.
NVIDIA describes a separate world-model role: generating controllable, physics-aware synthetic video from text, images, video, and sound. This is the data-generation function in its account—producing physical-world scenarios as well as reporting on observed ones.
When action-conditioned, Cosmos functions as a simulator, inferring actions and generating the next plausible futures for policy training and evaluation. The driving demonstrations show the action context NVIDIA has in mind. For construction cones encroaching on a vehicle’s lane, the displayed reasoning calls for a nudge to the right. For a vehicle crossing an intersection, it calls for yielding. Both examples pair that reasoning with a trajectory prediction labeled by time and lateral deviation.
Post-training ties planning to a particular body and task
NVIDIA’s adaptation claim is that post-training makes Cosmos a “world action model”: one that perceives, reasons, plans, and generates actions for an embodiment and environment.
The distinction appears in demonstrations involving materially different physical systems. A vehicle must respond to a rolling ball, lane encroachment, or cross traffic. A robotic arm must carry out a manipulation task in a constrained workspace.
In the arm demonstration, the instruction is to pick up a screwdriver and place it on the top rack. The displayed plan breaks the task into four actions: lower the gripper to grasp the screwdriver, lift it from the table, move it horizontally toward the rack, and place it on the designated shelf. The plan ties an instruction to a specific object, workspace, and robot configuration.
Cosmos is therefore presented as one post-trainable model spanning four related functions: observing physical scenes, generating synthetic video, simulating action-conditioned futures, and planning actions. The post-training step is what NVIDIA says connects those functions to the particular vehicle, robot, or other embodiment a developer is building.