World Models Aim to Replace Robotic Trial and Error

Ankit Gupta and Francois Chaubard place a practical constraint at the center of the robotics problem: the world contains abundant video of people and objects changing over time, but far less data that records the actions responsible for those changes. A video can show a chef making sushi or a car proceeding through traffic. It often cannot reveal the precise steering angle, brake pressure, joint command, grip force, or other control that produced the next frame.

That distinction matters because a controller needs more than an account of what usually happens. It needs to know what will happen if it acts. A policy answers: given the current state, what action should I take? A world model answers: if I take this action in this state, what state follows? In the discussion’s framing, the second capability is what allows an agent to simulate possibilities, create imagined experience for training, and plan around consequences instead of relying only on repeated physical trial and error.

The proposed opportunity is to start with passive data. A strong video model can learn broad regularities about how scenes evolve: hands move objects, vehicles navigate roads, ingredients are cut and assembled. It can then be action-conditioned with a smaller, more expensive dataset of state-action trajectories for a particular vehicle or robot. The resulting model could generate synthetic rollouts on which a policy is trained.

But the missing action labels are only one constraint. A controller also needs a predictive model accurate enough to plan with, an inference process fast enough for real-time control, and a way to recover when the world differs from its prediction. Those requirements explain why game-playing systems can use expensive search while driving and robotics remain difficult.

This is the context in which Gupta frames sample efficiency as one of AI’s largest unresolved problems: how quickly a system can learn a new skill from relatively few examples. Chaubard puts it beside “intelligence per watt,” or how much performance a system gets for a unit of energy expenditure. Intelligence per sample asks what one additional training example buys.

The relevant distinction is not merely whether a system can eventually acquire a skill, but how quickly it does so. Chaubard invokes François Chollet’s formulation of intelligence as a rate of skill acquisition. On tasks such as ARC-AGI, people can often infer a novel rule from a few examples and some thought, while frontier systems still struggle to generalize. Humans are not blank slates: they arrive with prior education, accumulated experience, and built-in biases. But the contrast remains important. A person who has encountered only a tiny share of the internet can sometimes solve a new problem from a handful of cases, while a model trained on internet-scale corpora may not.

The limiting case clarifies the ambition. Perfect sample efficiency would require no new samples from the live environment because a system could already predict the relevant consequences of action. Newtonian mechanics is the constrained practical example Gupta and Chaubard use: a spacecraft can be set on a trajectory to intercept an asteroid years in advance without learning from millions of failed launches. A sufficiently good model of the relevant physics can replace physical experimentation with calculation.

The wager is not that a useful world model must know everything. It must represent the parts of an environment that matter for a task well enough to replace some physical trial with internal simulation.

The strongest scaling constraint Chaubard identifies is action-space width. State spaces can be immense in both games and physical environments, but planning becomes impractical when an agent must seriously consider too many possible actions at every decision point.

Chess illustrates the distinction. The theoretical number of board configurations is vast, but the number of legal moves from a typical position is comparatively small. Chaubard uses roughly eight as a simplifying expectation. Go has a larger board and up to 361 possible placements, yet its action space remains bounded by stable game rules.

The AlphaGo-style approach combines a policy distribution over moves with a value estimate for reached states. Monte Carlo tree search, or MCTS, expands a selected subset of future branches. It balances exploitation—following moves that look valuable—with exploration of moves that have been visited less often. Rather than enumerate every possible future, it directs limited search toward branches the policy and value model consider promising while retaining coverage of alternatives.

The decision-time process is expensive even in this favorable setting. In Chaubard’s simplified account, AlphaGo runs 800 simulations to a depth of 30. That implies roughly 24,000 model invocations to select one move. The tree is not durable reasoning that can simply be reused on the next turn: once a move is made, the system discards the tree and builds another one. The resulting traversal distribution can be used as a training signal for the policy, but the inference cost remains.

For a hypothetical Go-like environment with one million possible actions, Chaubard estimates that maintaining roughly two samples per action would require two million simulations. At depth 30, that becomes roughly 60 million model invocations to choose a single action.

The arithmetic is illustrative, but the point is structural. Selective tree search only works when it can examine enough alternatives to be useful. It also benefits from fixed rules and enough time to deliberate. Go has both. A car or robot operates in a changing environment where the meaning of an action depends on other agents, and where a controller cannot wait tens of seconds or minutes before steering.

Chaubard identifies three limits on extending AlphaGo’s recipe. The action space has to remain small enough to search; the environment has to be sufficiently stable rather than changing as traffic, markets, and social settings do; and the system has to tolerate test-time planning latency. AlphaGo demonstrates what learned policies, learned values, and extensive search can do under unusually favorable conditions. It does not provide a general control method for embodied systems.

Ankit Gupta and Chaubard use self-driving to show why a modest-looking control interface hides a much harder problem. A car has steering, braking, and acceleration, but its state includes surrounding vehicles, camera observations, weather, road conditions, and the behavior of other people. They describe that state space as effectively infinite; the possible space of pixel observations alone is open-ended.

Deep learning has made such states more manageable by compressing them into learned latent representations. Game-playing systems do not enumerate every board state as a symbolic object; neural networks map board positions into representations that preserve useful patterns. Similar methods can compress camera streams and sensor data. Latent-space prediction is intended to make it cheaper to model the changes relevant to control without reproducing every visual detail.

Compression does not solve the central difficulty: other agents react. A car entering a roundabout or moving through dense traffic changes the behavior of nearby drivers. Chaubard argues that a rigidly Newtonian controller might decide the safest policy is never to move, because its prediction of everyone else’s unchanged trajectory would imply collisions. Functional driving requires modeling an environment in which other people adapt to the car’s own action.

The action space grows quickly even in the deliberately simplified calculation shown on screen. Dividing steering into 365 positions and brake and accelerator into 10 levels each yields 36,500 possible controls. That excludes timing, maneuver sequences, richer vehicle behavior, and the problem of anticipating how surrounding drivers will respond. The larger issue is not the precise number; it is that a controller chooses continually in an uncertain, sequential setting where each action changes the next state.

Gupta and Chaubard distinguish model-free and model-based reinforcement learning. A model-free system learns a direct mapping from observation to action:

A vision-language-action system can receive an image or sequence of images and output a next action. Chaubard compares this broadly to next-token prediction in language models: replace a tokenizer output with an action space, collect state-action demonstrations, and train the model to map state to action. With sufficient teleoperation data, this kind of behavior cloning can produce useful results. But it does not explicitly predict the transition connecting an action to its consequences.

A model-based system adds a transition model:

The policy selects actions while the transition model predicts the next state conditional on the current state and action. The benefit, in their account, is that the agent can train on simulated trajectories and, when time allows, plan by rolling possibilities forward. The cost is that planning is computationally expensive precisely where mistakes are costly and decisions must be immediate.

Francois Chaubard argues that robotics compounds the self-driving problem. Teleoperation data is expensive to collect and operationally difficult to scale. There is no direct equivalent of a large consumer vehicle fleet continuously generating action-labeled examples while people use a product.

A simple robot arm already illustrates the combinatorics. The sketch in the discussion totals 16 degrees of freedom for a basic arm and end effector. Giving each degree of freedom only 10 coarse action levels produces a joint space on the order of $10^{16}$ combinations. The calculation is not offered as an operational controller design. It makes the narrower point that a robot’s apparent set of simple motor commands becomes enormous when joints must be coordinated.

Robotics also has a cross-embodiment gap. A policy trained on one physical body may not transfer cleanly to another because mass, geometry, actuator response, and contact dynamics differ. Chaubard uses Tesla Model X and Model 3 vehicles as a simple comparison: braking, weight, and other dynamics differ enough that he would not assume data from one could be treated as interchangeable with data from the other. Differences among robot bodies can be larger still.

The proposed answer is to exploit human video where robot control data is scarce. Chaubard’s example is a sushi-making robot: record skilled chefs preparing sushi without instrumenting their hands, learn how the visual workspace changes, then introduce action conditioning with a smaller amount of robot-specific data. Society has substantial video of people performing tasks. It does not have a matching corpus of robotic state-action trajectories.

Chaubard traces the basic pattern to World Models, the paper by David Ha and Jürgen Schmidhuber shown during the discussion. He describes its central result as training a policy on imagined rollouts in environments such as car racing and Doom, then finding that the policy could perform in the actual environment. The important move was not simply predicting frames; it was treating a learned environment as a generator of synthetic experience for control.

He presents the Dreamer series, led by Danijar Hafner, as a major extension of that approach. A document shown on screen from DreamerV4 describes “imagination training inside the world model,” including an agent cutting down a tree, crafting a stone pickaxe, and mining diamonds in Minecraft. In Chaubard’s account, the system learns a world model, samples imagined trajectories inside it, and trains a policy from those trajectories.

The approach seeks to separate broad visual pretraining from task-specific action conditioning. First, a model learns state-to-next-state transitions from abundant passive video. Then it receives a smaller quantity of state-action data so that it can predict how an action changes the world. Finally, a policy trains against rollouts generated by that action-conditioned model.

Chaubard names video diffusion and flow-matching systems such as Sora and Wan as possible foundations for the first stage. He also points to NVIDIA’s World Action Models are Zero-Shot Policies, shown on screen under the name Dream2Real. Gupta describes that work as initializing from the open-source Wan video model and using roughly 500 hours of teleoperation data, with methods intended to operate across embodiments and unseen tasks.

The claim is not that passive video has solved robotics. A model may learn that hands, knives, ingredients, and tables change over time. It still has to learn which robot-specific commands cause those changes, how a new body differs from the body represented in training data, and what to do when a predicted outcome is wrong.

Francois Chaubard describes JEPA, or joint embedding predictive architecture, as a way to move prediction into latent space. Rather than reconstruct every pixel of the next image, an encoder maps observations into representations such as $z_t$. The world model predicts a future representation from the present representation and an action.

The attraction is computational and statistical. Predicting a compact latent can be cheaper than generating a full image, while a useful representation can omit visual details that do not matter for action. Chaubard links this strategy to latent diffusion and to earlier neural-network methods for compressing large state spaces before learning values or policies.

A naive latent-prediction objective can collapse. If an encoder and predictor produce trivial constant representations, predicting zeros could minimize a simple difference objective without learning meaningful structure. Chaubard names SigReg and VICReg as techniques intended to prevent this by placing distributional constraints on representations across a batch.

He also describes related experimentation in language models: predict the next token’s embedding rather than using a conventional cross-entropy output head over vocabulary tokens. In his account, this could act as a cheaper proxy objective because the cross-entropy head is expensive. He presents it as an active line of experimentation, not as an established replacement for standard language-model training.

The more consequential question is whether compressed representations can support physical prediction with sufficiently high fidelity. Chaubard is skeptical that current physics-informed neural networks, or PINNs, work well enough for this purpose. His example is a driving model trained overwhelmingly on normal road scenes. If asked to predict a rare trajectory toward a house, he argues that the model may transform the scene into something resembling the highway-like situations dominant in its data rather than faithfully predict a collision.

Rare safety-critical states cannot simply be diluted by ordinary examples. Chaubard argues that data mixing and minibatch composition have to preserve coverage of unusual cases. Even then, he says, learned models retain residual error. A model that predicts plausibly may still be inadequate when its forecast guides a sequence of physical actions.

His illustration is Stephen Curry reportedly noticing a dead spot on a basketball court from an unexpected bounce. The example concentrates the standard Chaubard has in mind: a useful model for dexterous control must distinguish between a merely plausible prediction and a physically reliable one.

He suspects the limitation is partly about stochastic gradient descent and partly about architecture. Transformers, in his view, do not compress information in the time domain in the way a sufficiently capable physical world model may require. He does not identify a replacement architecture. Fidelity remains an unresolved requirement rather than a matter of applying more of the existing recipe.

Francois Chaubard points to the speed of human adaptation in sports and control. A tennis player can adjust to a changed opponent; a person encountering a different surface can respond to changed friction with very little feedback. In robotics, a controller may need to estimate a changed friction coefficient immediately, rather than collect a new dataset and retrain. Chaubard says current systems do not yet have a convincing mechanism for this kind of test-time, out-of-distribution adaptation.

Real-time deliberation is a related problem. The full MCTS process used in AlphaGo cannot simply be inserted into driving or dexterous manipulation. Pretraining on imagined experience may place more capability in a fast policy, but maximum performance may still require some inference-time planning. The unresolved technical challenge is how to preserve the value of rollout-based deliberation without introducing unsafe latency.

Robots also lack much of the distributed sensory feedback humans use for contact-rich manipulation. Skin supplies information about texture, temperature, shear force, and friction across the body. A robotic system may have a limited tactile sensor, but not the pervasive multimodal feedback that lets a person determine whether an object is smooth, rough, slipping, or resisting motion. Chaubard says that numbing someone’s hands can make tying shoelaces impossible: vision still shows the laces, but it does not provide the same control loop.

A paper by Rajesh P. N. Rao shown on screen frames the neocortex as “active predictive coding,” in which cortical areas estimate latent sensory states and actions while predicting their consequences at multiple levels. Chaubard sees that description as closely resembling the joint state-action modeling used in model-based reinforcement learning. He treats it as support for the hypothesis that world modeling is central to biological intelligence, rather than proof that current architectures reproduce the brain’s mechanism.

That motivates his “squint test.” Airplanes did not need literal flapping wings to fly, he says, but they did require enough wing-like structure that the resemblance to birds is recognizable. By that standard, a system that models latent state, predicts action consequences, selects actions, and performs planning looks more brain-like to him than an autoregressive language model alone.

The analogy remains incomplete in his view. A brain appears to arbitrate among processes while acting, combine fast reactions with slower deliberation, and improve through offline consolidation. Chaubard points to sleep and hippocampal sharp-wave ripples as suggestive of a wake-sleep process: experience is collected during the day, replayed or compressed later, and used to update future behavior. He sees no clear equivalent of that full mechanism in current architectures.

For the next few years, Chaubard expects world models to become more central to robotics policies and possibly to more capable self-driving systems. But the path remains conditional. Synthetic experience helps only if the model’s predictions are reliable enough to train on, action conditioning transfers across bodies, inference is fast enough to control a machine, and the system can use feedback when the world departs from its expectations.