Hybrid Imitation Learning Builds Adaptive Parkour From 30 Seconds of Video

Károly Zsolnai-Fehér frames virtual parkour as a conflict between two capabilities that have been difficult to combine. A controller can be trained to reproduce human motion, yielding recognizable vaults, flips, and landings—but that approach is brittle and repetitive. Or it can be trained to reach a goal in an obstacle course, which makes it adaptive but can produce movement that no longer resembles an athlete’s.

The proposed answer, Hybrid Imitation Learning (HIL), is a single controller trained to do both: retain the human-like character of reference motion while departing from a script when a changing obstacle course requires it. Its inputs include information about the body, obstacles, and destination, so imitation is conditioned on the surrounding scene rather than treated as a sequence of poses to replay.

The training set is strikingly small. The paper’s parkour dataset contains 19 reference-motion clips and their corresponding obstacles, collected from YouTube—about 30 seconds of parkour in total. From those clips, HIL is shown learning a diverse set of parkour skills, composing them across longer courses, and adapting them to new obstacle configurations.

Previous methods illustrated the tension HIL is meant to address. One baseline, ASE, stumbles on a low obstacle rather than clearing it. AMP reaches a target by routing around an obstacle, a behavior that reaches the destination while evading the intended parkour task. MaskedMimic produces a more athletic-looking flip, but fails to land on the platform.

Those examples make an important distinction: task completion alone is insufficient, but motion realism alone is also insufficient. The desired behavior is an appropriate athletic action in context—jumping, vaulting, recovering, and choosing among skills as the course changes.

HIL is shown clearing blocks and platforms toward a target, then continuing through longer and more varied sequences. In demonstrations with randomized obstacle characteristics, the same skill is adapted to conditions beyond the training data. In another set of tests, it sequences multiple skills through obstacle placements it had not encountered before.

The paper also shows a controller trained on sequences of five obstacles being tested on much longer courses. Under the reported noisy setting, it achieves a 40% task-completion rate on sequences of 20 obstacles, with obstacle variation set at $\sigma =0.03$.

The mechanism is a hybrid training setup: one “classroom” teaches the controller to copy human movement, while another teaches it to solve new obstacle courses. Training happens simultaneously, with both settings providing the controller body, obstacle, and destination information.

Károly Zsolnai-Fehér explains the imitation component as a learned judge. The discriminator sees real human motion together with its obstacle context and is trained to classify it as real. It also sees motion produced by the controller in context and is trained to classify that as artificial. The controller then receives a better score as its actions become more human-like and suitable for the obstacle it faces.

The source describes three parts of that objective. One term rewards the judge for identifying real reference motion, conditioned on the surrounding obstacles, as real. A second rewards it for identifying controller-generated motion in context as artificial. A final stability term penalizes the judge for changing its decision too sharply in response to small changes in the motion-and-context input.

The controller and judge improve against one another: the controller learns to produce motions that the judge cannot easily distinguish from human ones, while the judge becomes more discriminating. That adversarial imitation signal is paired with task learning rather than replacing it. The controller is not simply asked to look human; it must navigate to a destination through obstacles. This is the source of HIL’s central promise: human-like movement that can still compose new actions in a course.

In the paper’s noisy, unseen-scene evaluation, obstacle position, orientation, and scale are perturbed. HIL posts the highest reported skill accuracy and the lowest tracking error among the listed methods, while retaining substantial task completion. But the table also exposes the trade-off emphasized in the source: a task-reward configuration achieves a higher completion figure, 0.86, than HIL’s 0.74, while HIL tracks reference motion much more closely.

The reported results pair a 0.31 tracking error and 0.66 skill accuracy for HIL with 0.74 task completion. On longer, noisier courses, the paper reports 40% completion on 20-obstacle sequences after training on sequences of five obstacles. The source also identifies another failure mode: after recovering from a stumble, the character can settle into an unnatural, glitch-like seated pose.

Yet the demonstrations go beyond the initial parkour course. The controller is shown in everyday interactions, including sitting on a chair, and executing a particularly complex maneuver: a wall run and flip between tall vertical blocks to cross a gap. These examples support a broader claim of generality, while the reported long-course completion rate and unnatural recovery motions mark clear limits.