Action-Based Search Could Make Web Video Usable for Robot Training

Rafael Levi’s central claim is that progress in robotics increasingly depends on finding better training data. Large language models have trillions of words to learn from, and image-generation systems have billions of labeled images. Robotics, Levi said, has only about a million videos of robots performing tasks. That scarcity makes it difficult to teach machines the range of actions and physical situations they will encounter outside a lab.

The usual response is to record demonstrations directly: ask people to open a door, sit in a chair, or perform another task for a camera. Levi argued that this produces a consequential bias. People behave differently when they know they are being recorded, and an action performed on cue may not resemble the same action in ordinary life. Someone asked to demonstrate opening a door might move slowly and deliberately, facing the camera; someone walking into a house may open it quickly, one-handed, without thinking about the movement at all.

That distinction matters because a robot trained on demonstrations needs to act in the world, not reproduce a performance staged for data collection. Levi’s point was not simply that hand-recording is expensive. It is that collecting more of the wrong kind of demonstration may not solve the underlying problem.

Levi contrasted staged demonstrations with video recorded in ordinary settings. Online video contains people handling objects, moving through spaces, and interacting with physical events. It shows gravity, motion, and cause and effect—the background knowledge a robot or world model needs in addition to any specific control instructions.

He described other common sources of training data as limited in different ways. Simulation can be cheap, but Levi said that simulated physics is not yet good enough to reliably transfer learned skills to the real world. Teleoperation—having a person control a robot by hand—can produce real robot data, but it is slow and tiring, and the hours that can be recorded do not scale easily. Existing datasets, he said, remain small.

The web appears to offer a much larger alternative. Levi pointed to the volume of video on services such as YouTube and Vimeo and asked how many clips show ordinary actions such as opening a door, pouring liquid, or folding clothes. The obstacle is that public video is not organized for this use. Searching by a video’s title will not reliably locate the action inside it, particularly when the title describes something else entirely.

Levi cited a Meta model as an example of how web video might support robot learning. He said the model was trained on about a million hours of public video and then required 62 hours of real robot footage to control a robot. The role of the web video, in his account, was to teach the model how the world works; the smaller amount of robot data supplied experience with the robot itself. He presented this as evidence that broad video pretraining can reduce, rather than eliminate, the need for robot-specific practice.

There is a limitation: ordinary video does not show how a robot moves or provide the sensor data used by a robot. A clip of someone pouring coffee does not directly tell a robot how to move its arm. Levi described a method that compares consecutive frames to estimate what changed: by measuring motion across a sequence, a model can infer features such as angles and distances. That information can help provide a basis for training, but Levi cautioned that downloaded video does not get a robot all the way to working behavior. It still needs processing, and some real robot data remains necessary.

A large video supply is useful only if relevant clips can be identified efficiently. Levi emphasized the waste involved in downloading broad collections and filtering them afterward. He cited NVIDIA’s Cosmos robot-data pipeline as discarding about 96% of the video it downloads, and Stable Video Diffusion as discarding 74%. In his framing, that means paying for bandwidth, storage, processing, and review before throwing most of the material away.

Levi also cited Stable Video Diffusion as discarding 74% of downloaded video. The two figures illustrate his concern: a pipeline that collects broadly and filters later can spend resources on footage that will not be used.

The proposed alternative is “search first, collect second.” Rather than download large quantities and identify useful footage later, a researcher describes the scene or action they want and retrieves matching clips. Queries can be specific: a person washing dishes by hand in a sink, removing grease with a sponge, rinsing plates, and placing them on a rack; or a person folding T-shirts, jeans, and towels on a flat surface, with different folding styles and speeds.

Bright Data’s Visual Index, Levi said, searches more than a billion web videos by what happens in them, rather than by their titles or other keywords. The demonstration showed a dishwashing search returning a grid of video thumbnails, followed by a search for clothes-folding clips. Levi acknowledged that results can include videos that are only partly relevant—for example, someone cleaning a kitchen rather than washing dishes. More detailed descriptions can narrow what comes back, but the results still need to be assessed for their match to the request.

The product’s purpose is to make a large, scattered video supply searchable by action and return clips rather than entire videos. Levi described the workflow as defining the scene in plain language, searching the indexed collection, and extracting trimmed clips prepared for further processing. The clips may still need additional work—for instance, processing for motion or distance information—before they are ingested into a robotics training pipeline.

Levi said the Visual Index is available through an API, not just through the interface shown in the demonstration. A request returns snippets of video and a link to the original source video. The response includes timestamps for the relevant moment, a score indicating how closely a clip matches the query, and a frame count for the portion that contains the requested action. Results can stream as they are found.

Those details matter because an action-oriented search is not the same as a guarantee that every result is suitable training data. A timestamp lets a user locate the relevant part of a longer video; the match score helps rank results; the source link lets the user inspect the original. Levi’s description treats the system as a way to target collection and reduce noise, not as a replacement for all downstream review or processing.

He argued that this reduction in noise could matter for training, where irrelevant data can create problems, including what he called hallucinations. The broader claim is that a system which retrieves a narrower set of clips can cut the cost of collection, storage, processing, and review while making it easier to build datasets around a particular action.

The same search capability, Levi said, could serve purposes beyond robotics. A company might want to find videos in which its product appears even when the video title has nothing to do with the product. His example was a makeup brand visible on a table while someone applies makeup in a video that is otherwise about a podcast or another activity. Searching for the brand in the title would miss that occurrence; searching by visual content could surface it. He also mentioned finding gameplay videos of people beating a particular level, or searching for other specific events inside video.

Levi’s examples for self-driving focused on actions that are specific to road behavior: a car stopping at a red light, running one, turning left or right, or encountering an unusual situation. He pointed to dashcam footage as a potential source of such examples, including the large volume of dashcam video he said is available online. In this case, the value is not simply that video shows physical events; it is that action-based search could locate particular driving situations inside clips whose titles may not describe them.

He also connected video search to world models that need to represent physical behavior: how objects fall, how people walk or sit, and how actions change the surrounding scene. These examples point to a broader use than robot manipulation or driving. Levi’s premise is that public video contains examples of physical interactions at scale, while current title-based search makes those examples difficult to find and collect selectively.

That leaves a practical distinction at the center of Levi’s proposal. Public video is not equivalent to robot-control data, and it does not remove the need for real-world robot practice. Its value, as he described it, is in providing broad exposure to natural actions and physical situations, then making it possible to retrieve relevant moments without first collecting and processing vast quantities of unrelated footage.