The world model that teaches a robot from video: V-JEPA 2

A robot picks up and moves an object after only watching video. Meta's V-JEPA 2 learns from a million hours of internet video with no action labels, then adds under 62 hours of robot video to pick and place objects zero-shot on two Franka arms, with no robot data and no reward ^[1]. The key is that it predicts the next scene in a learned representation space instead of drawing it in pixels. That made planning one action more than ten times faster than the video-generation approach ^[1].

This article reads the arXiv paper v1, its abstract, sections 3 and 4, Tables 2 and 3, and Figure 1, as of November 2026, checking the numbers against the source ^[1]. The paper is a June 2025 release, and the code and models are in the public repository. We did not reproduce the robot experiments, and we used only the values the paper states in text for robot success because the per-object table columns were ambiguous to extract. We have no relationship with the authors.

What is an AI world model?

A world model is an internal model that predicts how the next state changes when you act now. The idea comes from how people gather incoming information, predict what comes next, and plan from that prediction ^[1].

The hard part is how to learn it. Robot interaction data is scarce, while internet video is abundant but carries no labels for what action was taken. Earlier approaches went two ways. Building a predictive model from interaction data alone hits the data wall, and models that generate whole videos had only limited success at robot control ^[1]. V-JEPA 2 chose to learn how the world changes from plenty of general video first, then connect it to actions with a small amount of robot data.

What do you learn from video alone?

It splits into two stages. Stage one is self-supervised pretraining on a million hours of video and a million images with no action labels. The objective is not to reconstruct pixels but to predict the representation of a masked part ^[1].

Figure 1. V-JEPA 2 learns from internet video first, then adds 62 hours of robot video to build an action-conditioned world model. Prediction happens in representation space, not in pixels ^[1].

In the objective below, the visible input $x$ goes through an encoder $E_{\theta }$ into a representation, and a predictor $P_{\varphi }$ learns to match the representation of a masked future part $y$.

Here $\mathrm{sg}$ means no gradient flows to the target side, and $E_{\bar{\theta }}$ is a slowly trailing target encoder. Because the loss is a difference in representations rather than pixels, the model focuses on the parts useful for prediction instead of reconstructing every texture in the background ^[1]. The encoder scales from 300 million parameters (ViT-L) to 1 billion (ViT-g), and raising training iterations and resolution lifted downstream performance further ^[1].

How does it move a robot with no robot data?

Stage one alone cannot move a robot, because video has no actions. So stage two attaches a 300-million-parameter predictor on top of a frozen V-JEPA 2 encoder to build the action-conditioned world model V-JEPA 2-AC. Only this predictor trains on robot video from the Droid dataset: 23,000 trajectories including successes and failures, under 62 hours ^[1].

To move the robot, it is given a goal image and plans with model-predictive control. The model looks ahead at how an action changes the representation and picks the action that moves toward the goal representation. It uses a 7-axis Franka Panda arm, a two-finger gripper, and a single monocular RGB camera, and it ran zero-shot in two different labs with no robot data from those environments ^[1].

How well did it do?

On understanding and prediction it surpassed prior models built task by task.

Table 1. V-JEPA 2 on understanding, prediction and question answering. Action anticipation sets a new best over prior task-specific models ^[1].

On robots the balance of speed and success stands out. Pure reaching, moving the arm to a target pose, worked for every model compared. The gap opened on grasping and moving objects, and the paper states that V-JEPA 2-AC had the highest success rate across all tasks ^[1]. The video-generation baseline Cosmos reached 80% on reaching but was weak on object interaction.

The planning cost differs sharply. Cosmos generates pixels and took four minutes to choose one action, and more than an hour to finish a single pick-and-place trajectory. V-JEPA 2-AC predicts only representations, so even with ten times more candidates per step it took 16 seconds per action ^[1].

What is the difference between a video generator and a world model?

Even for the same future prediction, what you output matters. A video generator makes all the pixels of the next scene. It looks convincing but is expensive, and using it for control means drawing a video at every step. A joint-embedding predictive approach like V-JEPA 2 predicts only compressed representations, so the comparisons planning needs run fast ^[1].

The deeper question is whether drawing video well is the same as understanding physics. Other work takes this up separately. The Physics-IQ benchmark scored generators like Sora, Runway and Pika on physical principles such as fluid dynamics, optics and mechanics, and reported that visual realism and physical understanding are unrelated and that physical understanding is still far short ^[2]. That is less a rejection of video learning than a yardstick for what a model must pass to count as real robot intelligence.

There is also a push to get generalization from scale. GR-2 first learned how the world changes from 38 million internet video clips, then transferred to robot manipulation ^[3]. If V-JEPA 2 is the route through representation prediction and little robot data, GR-2 is the route through video generation and scale, two paths toward the same goal.

Limits

The paper is clear about what it cannot do.

• It is sensitive to camera position. The model infers the action coordinate axis implicitly from a monocular RGB video with no explicit calibration. When the robot base is outside the frame, the axis becomes ambiguous and prediction fails. The authors manually tried several camera positions before settling on one ^[1].
• Long-horizon planning is hard. Predicting many steps autoregressively lets error accumulate ^[1].
• It centers on grasping. It handled prehensile tasks that can be set with a goal image, and used no reward or task-specific training. That is both a strength and a boundary ^[1].

To this we would add an open question of our own: the variance in per-object success is tangled up with camera placement and gripper control precision, so a single success rate is a poor gauge of the model's physical understanding.

In practice

This is not a paper that overlaps directly with what TRAIL does. Still, one design principle reads as the same family. Just as V-JEPA 2 predicts in representation space instead of making every pixel to plan faster, content generation also holds consistency and efficiency more easily when it works in a space that compresses a brand's features rather than imitating the whole output.

More broadly, it shows the terrain where generative models widen from making text and images toward action and control. The asymmetry of learning the world from video first and tying it to actions with little interaction is a design that borrows knowledge from the data-rich side for the data-poor side, worth revisiting in content automation too.

Read next

This is the first article in the line of our paper reviews that covers video and generative models. For how video understanding is used in retrieval, see the video RAG paper review; for how search handles a user's question, see the analysis of AI search queries; and for using representations and features in content, see the structural feature engineering article.