V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
1. Paper overview
In one sentence: Video feature prediction can support robot planning after action-conditioned post-training, but the demonstrated transfer depends on image subgoals, short planning horizons and camera placement. e02e03e07e08e09e12e18e24
| At a glance | What to know |
|---|---|
| Research problem | Source description Can abundant action-free video supply representations useful for understanding and control, while scarce robot interaction data teaches the effects of actions? The authors pursue prediction in learned feature space to avoid the cost of generating pixels during planning. Observation pretraining alone does not specify how a robot command changes a scene. e02e03e07 |
| Core mechanism | |
| A key reported result | Zero-shot cup and box pick-and-place: V-JEPA 2-AC: cup 80%, box 65%, averaged across labs. Task success rate. Two new labs; closed-loop image-goal control, two intermediate subgoals, ten trials per task in each lab. DROID-finetuned Octo: cup 15%, box 10%. V-JEPA 2-AC box success is 80% in Lab 1 and 50% in Lab 2. Physical execution demonstrates transfer under this setup. Single-goal grasping remains weaker: average cup 65% and box 25%. These distinct tasks should not be treated as a common difficulty scale; no confidence intervals are reported. e10e11e12 |
| Reading caution | Source description Camera placement is a substantive condition of transfer: the authors manually selected a favorable position. Appendix B.4 measures camera-dependent rotation errors in the inferred action axes and explicitly says its proposed calibration was not used in the reported experiments. e14e24 |
Core contributions
Figure 2. Observation pretraining supplies the representation; robot post-training learns how actions change it. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each panel from the images upward. On the left, the unmasked clip supplies the target branch, while the Mask block removes patches before the trainable encoder processes the visible clip. Mask tokens tell the predictor which missing locations to predict. The dotted EMA arrow updates the target encoder, and the stop-gradient marker blocks gradients through its targets. On the right, both encoder boxes are frozen: past frames become predictor inputs and the future frame supplies the target. Robot actions and poses replace mask-location conditioning. The caption specifies block-causal attention, allowing current and earlier tokens; the right panel therefore represents a separately learned transition model. e03e04e07e08
What it supports. The transfer interface is a learned feature space: the robot predictor learns to forecast the frozen encoder's representations. This preserves the observation-trained representation during robot adaptation. The L1 boxes compare features in both stages, while their conditioning information and training targets differ substantially.
Where the evidence stops. The two predictor boxes are different trained modules. The action-free pretraining predictor has 22M parameters, whereas the robot predictor has approximately 300M. The architecture alone does not establish accurate dynamics for arbitrary actions or viewpoints.
2. Motivation
2.1 The problem and the proposed response
Can abundant action-free video supply representations useful for understanding and control, while scarce robot interaction data teaches the effects of actions? The authors pursue prediction in learned feature space to avoid the cost of generating pixels during planning. Observation pretraining alone does not specify how a robot command changes a scene. e02e03e07
2.2 What this reading follows
V-JEPA 2 starts with videos that contain no robot commands and learns to predict missing visual representations. To control a robot, the authors freeze that representation and train a new predictor on synchronized robot video and state changes. A planner then searches for actions whose predicted outcome resembles a goal image. This reading follows the separation between learning features, learning action effects and selecting commands. It also connects the strong anticipation results to an informative ablation: observed features already explain most of that benchmark score. The camera diagnostic supplies a complementary boundary on the physical transfer demonstrated in the robot tables. e02e03e07e08e09e12e18e24
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded Dual-system / Other mechanisms / Outside quadrants assessment is consistent with the deployed architecture: a frozen perception encoder and separately trained forward dynamics predictor feed an external CEM planner and controller. Actions condition future-state prediction and are optimized at inference; they are not jointly emitted with futures or extracted by a learned inverse-dynamics head. Joint encoder/predictor pretraining does not establish a unified action-generating model. The WAM and latent-prediction tags fit this explicit inference-time world-model use. e03e07e08e09e11
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Learn a representation before asking it to predict commands
The first training stage receives observations without robot actions. Its prediction problem is spatial and temporal mask completion in representation space: visible video tokens must support predictions of the missing tokens' EMA-encoder features. Stop-gradient and the moving target encoder are central to that learning setup. They prevent the target branch from simply following the instantaneous gradient through the prediction loss. Larger encoders, a curated mixture of video and images, and longer clips during cooldown extend this recipe. The robot stage then keeps the encoder fixed, so its state representation cannot adapt to the robot loss. What transfers is the learned visual coordinate system for describing scenes. The separately trained action-conditioned predictor must learn how commands move a scene through that representation space. e03e04e05e06e07e08
Figure 7. Robot commands are selected by searching through predicted consequences. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the current observation on the left. Its encoder output enters the predictor, which is reused along a candidate action sequence. The bell-shaped graphics under the predictor blocks depict the action proposals optimized by CEM; they are not generated video frames. On the right, the goal image is encoded into the same feature space. The red L1 comparison measures the discrepancy between the final predicted representation and the goal. Equation (5) also conditions on the current robot pose, although the schematic omits that input. CEM refines action distributions from elite candidates while model weights stay fixed. The text then closes the loop by executing one action and observing again. e09e10e11e30
What it supports. The drawing permits a general rollout horizon, but the reported robot experiments use horizon one, 800 candidates and ten refinement iterations. Pick-and-place succeeds through repeated feedback and externally supplied subgoals switched after 4, 10 and 4 steps. A multi-block schematic should therefore not be read as evidence of long-horizon experimental planning.
Where the evidence stops. The action-bound description remains ambiguous: p. 12 pairs an L1 radius of 0.075 with roughly 13 cm maximum displacement. Exact replication needs that convention resolved. The displayed L1 objective is a feature distance, not a measured task-success reward.
5.2 Turn robot state changes into a feedback controller
DROID contributes synchronized frames and end-effector states, including orientation and gripper position. The training actions are changes between adjacent states; the predictor consumes those actions, poses and encoded images with causal attention. Teacher forcing teaches transitions from observed states, while the two-step recurrent loss exposes the predictor to its own intermediate prediction. At deployment, an image supplies the desired visual state. Candidate actions are scored by the predicted final feature distance to that goal. CEM repeatedly concentrates its proposals around low-energy candidates, and the low-level controller executes the first selected command before the camera supplies new feedback. In the reported setup, planning looks only one step ahead. The grasp, transport and placement images supplied by the experimenter provide the decomposition needed to compose those local decisions into pick-and-place behavior. e07e08e09e10e11
5.3 Ask what each experiment says about a world model
Reader analysis: the paper's evidence should be read as several complementary tests. SSv2 shows that useful motion information is recoverable by a trained probe. EK100 asks about future human-action categories, yet the input ablation shows that encoder features already deliver almost all of the reported action recall. Video QA adds language alignment and substantial supervised data, so its answers cannot be attributed to the action-free backbone alone. Robot execution is the direct test of action-conditioned planning, and its successes therefore matter beyond visually plausible predictions. Those successes still occur with image subgoals, selected viewpoints and small trial sets. The camera diagnostic links a concrete representation failure to the control setup: a scene can be recognizable while the model misinterprets which robot-coordinate direction an action will move. e12e14e15e18e20e22e24
5.4 Training and inference
During training
VM22M mixes SSv2, Kinetics, HowTo100M, curated YT-Temporal-1B and ImageNet. Curation retrieves scene clusters using training-set target distributions and excludes target validation videos from the candidate pool. Training starts with 16-frame, 256-pixel clips and increases duration to 64 frames during cooldown; 384- and 512-pixel variants are documented. e05e06e26
Robot post-training retains under 62 hours of DROID left-camera video after discarding clips shorter than four seconds. Clips contain 16 frames at 4 fps and 256 × 256 resolution. 'Unlabeled' excludes reward, task and success labels, but retains synchronized robot state signals and both successful and failed trajectories. e07e10e23
The AC optimizer is AdamW with batch 256 and weight decay 0.04. Learning rate warms from 7.5 × 10^-5 to 4.25 × 10^-4 over 4,500 iterations, stays constant for 85,500 and decays to zero over 4,500. e23
During inference
Encode the current and goal images, roll out candidate actions and minimize final-feature L1 distance to the goal using the cross-entropy method. Execute only the first action through the low-level controller, obtain fresh visual feedback and replan. Deployment uses blocking control, an uncalibrated monocular camera and Franka arms with RobotiQ grippers. e09e10
The reported robot setup uses horizon one, 800 candidates, ten refinement iterations and ten elite samples. Pick-and-place supplies grasp, transport and placement images, switching after 4, 10 and 4 controller steps. This is an externally decomposed task, rather than autonomous discovery of subgoals. e11
5.5 Implementation flow
- Learn a predictable visual representation
Patchify clips into 2 × 16 × 16 tubelets. A ViT encoder processes visible tokens, and a predictor combines their embeddings with location-bearing mask tokens. Its masked-patch predictions match an exponential-moving-average encoder's targets with L1 loss and stop-gradient. The largest encoder is a 1B-parameter ViT-g; the pretraining predictor is 22M parameters. e03e04
- Make the representation action-conditioned
Freeze the encoder and encode robot frames independently into 16 × 16 × 1408 feature maps. Interleave each map with a seven-dimensional pose and action: Cartesian position, extrinsic Euler orientation and gripper state, or their changes. A new approximately 300M-parameter predictor uses 24 layers, 16 heads and width 1024. Block-causal attention permits current and past tokens, excluding future frames. e07e08
- Learn both observed-state and recurrent transitions
Teacher forcing predicts the next representation from observed history. A second loss feeds predictions back into a two-step rollout. Only predictor weights update. Figure 6 illustrates four steps, but the implemented rollout loss uses two; these should not be confused. e08
- Attach task-specific readouts
Classification trains a four-block attentive probe over frozen encoder features. Anticipation combines frozen encoder features with predicted future features and uses separate verb, noun and action classifiers trained with summed focal losses. Video QA instead aligns visual tokens with an LLM through a projector; controlled experiments explore frozen and unfrozen encoders. e15e16e17e20
6. Experiments & results
V-JEPA 2 learns video features through masked representation prediction. A separate action-conditioned predictor uses the frozen encoder to support image-goal planning on real robot arms. Classification probes, anticipation probes and language alignment test complementary capabilities; robot control remains dependent on subgoals and camera placement.
6.1 Read the original evidence
Table 2. Cross-lab execution improves strongly, with substantial object and task dependence. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First choose the task column, then compare methods within the same lab. The italic Avg rows summarize the two labs; they are not additional experiments. Every task has ten trials per lab with variations such as object position and starting pose. Reach has no object column, whereas the remaining skills separate cups and boxes. Grasp and reach-with-object use one goal image. Pick-and-place receives two intermediate image goals plus the final goal, so its larger success rate should not be interpreted as proof that it is an intrinsically easier task. The table measures closed-loop physical execution after robot-data training, with no training on either evaluation lab. e10e11e12e14
What it supports. Average pick-and-place success is 80% for cups and 65% for boxes, compared with Octo's 15% and 10%. Yet average single-goal box grasping is only 25%, and box pick-and-place varies from 80% in Lab 1 to 50% in Lab 2. The result supports useful transfer while exposing substantial residual failure.
Where the evidence stops. These small trial sets have no reported confidence intervals. Octo and V-JEPA 2-AC have different training histories, and camera placement was manually selected. The comparison supports the reported systems under these conditions, without isolating the effect of JEPA pretraining.
Table 5. A frozen video backbone supports strong one-second human-action anticipation. Original paper, p. 17 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the Action column for the combined action-category benchmark; the Verb and Noun columns report separate classifiers. The metric is mean-class recall-at-5, so it asks whether the annotated category appears among five proposals and averages across classes. The lower block uses frozen V-JEPA 2 features with a supervised anticipation probe. Encoder features describe the context, while the action-free predictor supplies features for a future frame. Context ends one second before the action. The ViT-g384 row uses 32 frames at 8 fps and 384-pixel resolution. Compare it with the ordinary ViT-g row to see the resolution change, then inspect Table 20 to isolate feature inputs. e16e17e19
What it supports. The largest high-resolution model reaches 39.7 action recall-at-5 versus PlausiVL's 27.6: a 12.1-point increase, approximately 44% relative. Even the ViT-L row reaches 32.7. Verb and noun scores also improve, but their values describe different classification problems and should not be averaged into a robot-control success rate.
Where the evidence stops. These methods do not share matched pretraining data or compute. EK100 is a kitchen benchmark with a fixed action vocabulary, and the paper reports degradation at longer anticipation times. Strong one-second category prediction does not establish open-ended forecasting or robot execution.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Zero-shot cup and box pick-and-place Two new labs; closed-loop image-goal control, two intermediate subgoals, ten trials per task in each lab. | V-JEPA 2-AC: cup 80%, box 65%, averaged across labs. Task success rate | DROID-finetuned Octo: cup 15%, box 10%. V-JEPA 2-AC box success is 80% in Lab 1 and 50% in Lab 2. Physical execution demonstrates transfer under this setup. Single-goal grasping remains weaker: average cup 65% and box 25%. These distinct tasks should not be treated as a common difficulty scale; no confidence intervals are reported. e10e11e12 |
| Lab 2 planning latency and manipulation Single NVIDIA RTX 4090; ten CEM refinements, horizon one; 800 samples for V-JEPA 2-AC versus 80 for Cosmos. | 16 seconds per action; cup 80%, box 50%. Planning time per action; pick-and-place success | Cosmos: four minutes per action; cup 0%, box 0%. Both obtain 20% box-grasp success. This is a system-level latency and control comparison with unequal candidate counts and different pretrained models, not an isolated objective ablation. The table includes a tie despite broader prose claiming superiority across skills. e10e13 |
| SSv2 motion classification Frozen ViT-g384 encoder and supervised attentive probe; validation, 384 × 384, 64 frames × two temporal crops × three spatial crops. | 77.3% Top-1 accuracy | V-JEPA 2 ViT-g at 256 pixels/16 frames: 75.3%; InternVideo2s2-1B under the standard evaluation: 69.7%. The strongest score also increases evaluation resolution and duration. Pretraining data differ across encoders, so comparisons establish system performance rather than causal superiority of one learning objective. e15 |
| EK100 one-second action anticipation EK100 validation; frozen backbone with supervised probe; 32 context frames at 8 fps, 384 × 384; context ends one second before the action. | 39.7 / 63.6 / 57.1 Mean-class recall-at-5 for action / verb / noun | PlausiVL action recall-at-5: 27.6; the action gain is 12.1 points, approximately 44% relative. Strong category anticipation within cooking videos; this score does not measure robot-command generation or executed task success. e16e17 |
| EK100 predictor-input ablation Appendix D.2 changes which frozen features feed the anticipation probe. | Encoder plus predictor: 39.7. Action mean-class recall-at-5 | Encoder alone: 39.1; predictor alone: 20.2. The incremental gain from predicted features is 0.6 points. Most of this benchmark performance is already supported by encoder features; seed variation and uncertainty are not reported. e18 |
| Video QA with frozen encoders 18M alignment samples, Qwen2-7B-Instruct, frozen encoders; controlled alignment/evaluation with encoder-dependent resolutions and pooling. | V-JEPA 2 ViT-g512: 52.3. Reported mean across seven benchmark scores | Perception Encoder: 49.1; SigLIP2: 48.1; DINOv2: 45.7. Language-free video pretraining can supply useful visual tokens for language alignment. The aggregate combines different benchmark metrics; V-JEPA 2 slightly trails PE and SigLIP2 on PerceptionTest validation. e20e21 |
| Scaled video question answering 88.5M alignment samples; ViT-g384 with Llama 3.1 8B; 32-frame evaluation. PerceptionTest uses task-specific SFT and its test split; MVP is zero-shot. | 84.0; 44.5. PerceptionTest accuracy; MVP paired accuracy | PLM 8B: 82.7; 39.7. Table 8 also reports leading results on TempCompass, TemporalBench and TOMATO in its comparison, but weaker TVBench and MVBench results than PLM. This is not a pure data-scaling ablation: backbone, projector and evaluation settings differ from the controlled experiment. e22e28 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 20. Predicted features add a small increment to already strong observed features. Original paper, p. 44 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The checkmarks identify which token sources reach the anticipation probe. The first row uses the encoder's observed-context features; the second uses only the action-free predictor's future features; the last combines both. Read horizontally to compare verb, noun and action performance for one input choice, then vertically within a column to assess the change. This predictor belongs to the video anticipation pipeline, not the robot action-conditioned model. The classifications remain supervised even though the backbone is frozen and was pretrained without labels. The cleanest question answered here is whether predicted tokens add useful information to the probe beyond the encoder tokens already available. e16e17e18
What it supports. Combined features improve action recall-at-5 from 39.1 to 39.7, a 0.6-point gain. Verb recall rises from 61.3 to 63.6, while noun recall rises only from 57.0 to 57.1. Predictor-only action recall is 20.2. This supports a modest complementary signal, with most anticipation performance already accessible from observed-context features.
Where the evidence stops. No seed variation, uncertainty interval or matched-token-count control accompanies these rows. This is not an ablation of robot rollout training, and it cannot establish that forward prediction is dispensable for the MPC controller.
Figure 16. Visual transfer does not remove the need to infer the robot's coordinate frame. Original paper, p. 40 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the horizontal axis as camera position around the robot: the caption defines zero degrees at the robot base and ninety degrees to its left. The vertical axis is the rotation error of the inferred action coordinates in the horizontal plane, not Cartesian reaching error. For each camera position, the authors collect a 201-step random-motion trajectory, infer actions between adjacent frames and compare them with real actions. A least-squares map between the 200 action pairs is decomposed to obtain a rotation component. The seven plotted points summarize this diagnostic. Their overall increase, with local reversals, shows that the same model's action interpretation depends on the viewpoint. e14e24
What it supports. The error is approximately five degrees near a 37-degree camera position and exceeds thirty degrees near 73 degrees. These approximate readings illustrate the overall viewpoint sensitivity, not a precise fitted law. The diagnostic helps explain why manually choosing a favorable camera position matters for the zero-shot manipulation results.
Where the evidence stops. The analysis concerns planar action axes and a limited set of camera positions. It does not measure success at every viewpoint. The authors propose a correction from random-action data but explicitly state that no such calibration was applied in their reported robot experiments.
7. Analysis & limitations
7.1 What the evidence leaves open
Camera placement is a substantive condition of transfer: the authors manually selected a favorable position. Appendix B.4 measures camera-dependent rotation errors in the inferred action axes and explicitly says its proposed calibration was not used in the reported experiments. e14e24
Autoregressive error and action-search complexity limit long-horizon planning. The robot tasks use image goals and scheduled subgoals; language-conditioned robot control and subgoal-free complex manipulation remain future work. e11e14
EK100 uses a closed vocabulary in kitchens, and anticipation degrades at longer horizons. Decoded robot predictions are qualitative diagnostics produced by an auxiliary decoder; their visual plausibility cannot substitute for execution success or a general test of physical understanding. e19e25
7.2 Questions for discussion
- Would the small EK100 predictor-feature gain persist with matched token counts, multiple probe seeds and longer anticipation times?
- How much of cross-lab control depends on selecting a favorable camera position?
- Can image subgoals be removed while maintaining success under a fixed planning-time budget?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs the pretrained encoder, DROID state/video synchronization, framewise encoding, left-camera filtering, pose/action conventions and the AC optimizer recipe. The full pretraining route additionally needs the VM22M mixture and retrieval-based curation. The two proposed checks in the illustrated edition target predictor utility and camera sensitivity without claiming a full reproduction. e04e05e07e23e24e26
Language alignment is expensive: the controlled setup reports 128 H100 GPUs; scaled Stages 2–3 report 512 H100 GPUs. Appendix E specifies PyTorch 2.5.1, with Transformers 4.46.0, Flash Attention 2 and DeepSpeed 0.14.4 for the controlled setup. These are reported requirements, not verified installations. e28
Resolve source discrepancies before exact replication: Section 2.4 gives 12K warmup + 228K constant + 12K cooldown, whereas Figure 5 describes 252K before an additional 12K cooldown. Appendix E.3 says Stage-3 batch 1024, but Table 22 says 2048. Table 4 says 384-pixel ImageNet evaluation for ViT-g384; Appendix C.1 says 512. The action-bound description pairs an L1 radius of 0.075 with approximately 13 cm displacement, leaving its implemented constraint unclear. e27e28e15e30
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does the predictor add forecasting information beyond extra probe tokens?
Reader-proposed check, not performed: freeze one ViT-g384 checkpoint and train the same four-block EK100 probe in three conditions: encoder-only, encoder-plus-predictor, and encoder features duplicated to match the combined token count. Keep the 32-frame, 8-fps context, training anticipation randomization, optimizer search and training budget fixed; repeat across seeds. Evaluate mean-class action recall-at-5 on the official one-second validation protocol, with longer anticipation reported separately. A stable gain for predicted features beyond both controls would support useful forecast-specific information. If duplicated encoder tokens match the gain, additional readout capacity or token count becomes a competing explanation for Table 20. e16e17e18e19
Check 2: Can a fitted action-coordinate correction improve held-out control?
Reader-proposed check, not performed: reproduce the planar random-action diagnostic at several camera positions and split each view's inferred/real action pairs into fitting and held-out subsets. Compare uncorrected commands, the fitted planar correction and a deliberately mismatched-view correction. Keep model weights, image goals, controller and CEM budget fixed. Measure held-out action error and repeated cup/box task success, reporting failures and calibration cost. Improvement in both measures would support coordinate misalignment as a practical bottleneck; lower action error without better grasping would expose additional contact or gripper failures. Label corrected runs as using local calibration data, since they change the original zero-shot deployment condition. e10e11e12e14e24
8.3 Reading coverage
Visual audit: Visually inspected the title and author block, all main-body figures and tables, and every appendix page containing training, evaluation or diagnostic material (pp. 1–23 and 33–48). Cross-checked the EMA/stop-gradient branches, frozen encoders, causal conditioning and planning objective against the text and equations. Figure 6's illustrative four-step rollout was distinguished from the two-step training loss. Inspected all six final crops individually; the two narrow anticipation tables were rendered at higher DPI, without upscaling. References on pp. 24–32 were read in the complete text pass. External code and separate supplements were outside this pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, author block, abstract and Section 1 Introduction (pp. 1–3)
- Section 2, including 2.1–2.4: objective, scaling, data and pretraining recipe (pp. 4–8)
- Section 3, including 3.1–3.2: action-conditioned training and planning (pp. 8–11)
- Section 4, including 4.1–4.3: robot setup, results and limitations (pp. 11–15)
- Sections 5–6: classification and action anticipation (pp. 15–18)
- Section 7, including 7.1–7.4: video question answering (pp. 19–21)
- Sections 8–9, acknowledgements and references (pp. 21–32)
- Appendix A.1–A.4.3: pretraining, curation, architecture and scaling ablations (pp. 33–37)
- Appendix B.1–B.4: robot hyperparameters, task definitions, decoded predictions and camera sensitivity (pp. 37–41)
- Appendix C.1–C.2: classification protocols and ablations (pp. 41–43)
- Appendix D.1–D.2: anticipation protocols and diagnostics (pp. 43–45)
- Appendix E.1–E.3: language alignment, software, hardware and evaluation (pp. 45–48)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All 15 supplied text chunks were read individually. The retained PDF was visually inspected on pp. 1–23 and 33–48; references on pp. 24–32 were read as text.
- Identity/version note: the title and identifier match. The supplied PDF displays arXiv:2506.09985v1, 11 Jun 2025, while its internal title-block date is June 13, 2025. Both dates are preserved; no later edition was consulted.
- Author metadata discrepancy: the PDF credits Mahmoud Assran where the catalog says Mido Assran, and credits Mojtaba Komeili as one author where the catalog splits Mojtaba and Komeili. The remaining author sequence agrees; metadata uses the observed title-page credits.
- The linked code, datasets and external supplements were not opened, and no experiments were reproduced. Code availability and installation were not independently verified.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author/affiliation block and arXiv margin
Exact title matches; margin identifies 2506.09985v1 dated 11 Jun 2025, internal date is June 13, 2025. Credits include Mahmoud Assran and the single author Mojtaba Komeili. Affiliations are FAIR at Meta and Mila – Quebec AI Institute and Polytechnique Montréal.
Go to primary source ↓e02PDF p. 2, Figure 1 and Introduction
Observation pretraining branches into attentive probing, language alignment and action-conditioned robot post-training; the latter supports MPC.
Go to primary source ↓e03PDF p. 4, Figure 2, Section 2.1 and Eq. (1)
Masked-patch L1 regression targets an EMA encoder with stop-gradient. After pretraining the encoder is frozen for action-conditioned prediction.
Go to primary source ↓e04PDF p. 5, Section 2.1 Architecture; p. 35, Appendix A.3, Table 12
Tubelets are 2 × 16 × 16 with 3D-RoPE. ViT-g has about 1B parameters, width 1408 and 40 layers; the action-free predictor is a 22M-parameter ViT-s.
Go to primary source ↓e05PDF p. 6, Section 2.3 and Table 1
VM22M combines SSv2, Kinetics, HowTo100M, curated YT-Temporal-1B and ImageNet, with source-specific weights. Target validation videos are excluded from the initial curation pool; images are temporally repeated.
Go to primary source ↓e06PDF pp. 7–8, Section 2.4; p. 33, Appendix A.1, Table 9
Pretraining uses progressive duration/resolution and a warmup-constant-cooldown schedule. Table 9 gives 16 to 64 frames, 4 fps, global batch 3072 and cooldown crop sizes 256/384/512.
Go to primary source ↓e07PDF p. 9, Section 3.1, Model inputs and Loss function
Under 62 hours of DROID video remain after filtering short clips. Inputs use 4-second, 16-frame clips at 4 fps; 7D state/action signals are retained. Frozen ViT-g produces 16 × 16 × 1408 frame features.
Go to primary source ↓e08PDF p. 9, Eqs. (2)–(4); p. 10, Figure 6 and Architecture
Teacher forcing uses 15 transitions; rollout loss uses T = 2 although the diagram illustrates four steps. The new approximately 300M predictor uses 24 layers, 16 heads, width 1024 and block-causal attention.
Go to primary source ↓e09PDF pp. 10–11, Section 3.2, Eq. (5) and Figure 7
CEM optimizes candidate actions by final predicted-to-goal feature L1 distance. Gaussian proposals are refined from elite trajectories, their means determine the selected sequence, and only the first action is executed before replanning.
Go to primary source ↓e10PDF pp. 11–12, Section 4.1, baselines, footnote 1 and Robot deployment
Octo is finetuned on all DROID; V-JEPA 2-AC uses 23k trajectories including successes and failures. Cosmos is separately finetuned. Evaluation uses Franka arms with RobotiQ grippers in two labs absent from DROID, monocular RGB and blocking world-model control.
Go to primary source ↓e11PDF p. 13, Section 4.2 Prehensile manipulation; pp. 37–38, Appendix B.2 and Figure 14
Pick-and-place receives two subgoals plus a final goal, switched after 4/10/4 steps. B.2 specifies horizon one, 800 samples, ten refinements and ten elites.
Go to primary source ↓e12PDF p. 14, Table 2, V-JEPA 2-AC and Octo rows, Grasp and Pick-&-Place columns
Ten trials per task per lab. Average V-JEPA 2-AC cup/box pick-and-place is 80%/65%, versus Octo 15%/10%. Box rates are 80% and 50% across labs; average cup/box grasp is 65%/25%.
Go to primary source ↓e13PDF p. 13, Section 4.2 planning comparison; p. 15, Table 3
Single RTX 4090, horizon one and ten refinements. V-JEPA 2-AC uses 800 samples and 16 seconds/action versus Cosmos 80 samples and four minutes/action. Lab 2 pick-and-place is 80%/50% versus 0%/0%; box grasp ties at 20%.
Go to primary source ↓e14PDF pp. 14–15, Section 4.3; p. 23, Section 9 Future work
The authors disclose manually selected camera placement, accumulating rollout error, growing planning complexity and reliance on visual goals/subgoals; language goals and more capable long-horizon models remain future work.
Go to primary source ↓e15PDF pp. 15–16, Section 5 and Table 4; pp. 41–42, Appendix C.1 and Tables 15–16
Frozen attentive-probe classification. SSv2 ViT-g384 achieves 77.3 at 64 × 2 × 3 and 384 pixels; ViT-g standard score is 75.3, InternVideo2s2-1B 69.7. Training data differ. ImageNet resolution differs between Table 4's caption and C.1.
Go to primary source ↓e16PDF pp. 17–18, Section 6 and Table 5
EK100 validation action anticipation uses one-second anticipation. ViT-g384 reaches action/verb/noun mean-class recall-at-5 of 39.7/63.6/57.1; PlausiVL action score is 27.6. Encoder and predictor features feed independent supervised classifiers.
Go to primary source ↓e17PDF pp. 43–44, Appendix D.1 and Table 19
Anticipation context is 32 frames at 8 fps, 384 pixels for ViT-g384. Probe training randomizes anticipation time 0.25–1.75 seconds and point 0–0.25, while validation uses one second and point zero; focal-loss parameters are specified.
Go to primary source ↓e18PDF p. 43, Appendix D.2 Impact of Architecture; p. 44, Table 20, three input configurations
Encoder-only action/verb/noun scores are 39.1/61.3/57.0; predictor-only 20.2/48.7/34.7; combined 39.7/63.6/57.1. No uncertainty estimates accompany the rows.
Go to primary source ↓e19PDF p. 18, Section 6 Limitations; pp. 44–45, Appendix D.2 and Figure 18
Kitchen-only closed-category evaluation limits generalization; performance declines with longer anticipation times. Verb, noun and action classifiers can disagree.
Go to primary source ↓e20PDF pp. 19–20, Sections 7.1–7.3; p. 46, Appendix E.2
Visual tokens are projected into an LLM. Controlled alignment uses 18M pairs and Qwen2-7B-Instruct, exploring frozen versus unfrozen encoders. E.2 additionally describes a high-quality image-captioning Stage 1.5.
Go to primary source ↓e21PDF p. 20, Tables 6–7; pp. 46–47, Appendix E.2 and Table 21
Frozen-encoder benchmark averages are 52.3 for V-JEPA 2, 49.1 for PE, 48.1 for SigLIP2 and 45.7 for DINOv2. PerceptionTest is validation after SFT. Encoders differ in native resolution and pooling; controlled inference samples 128 frames.
Go to primary source ↓e22PDF p. 21, Table 8 and Section 7.4; p. 48, Appendix E.3 Evaluation
Scaled alignment uses 88.5M pairs and Llama 3.1 8B. V-JEPA 2 scores 84.0 PerceptionTest test accuracy after SFT and 44.5 MVP paired accuracy, versus PLM 82.7/39.7. Other listed tasks are zero-shot; V-JEPA 2 trails PLM on TVBench/MVBench. Evaluation uses 32 frames.
Go to primary source ↓e23PDF p. 37, Appendix B.1 Post-Training Hyperparameters
AC training uses left-camera DROID clips, AdamW, weight decay 0.04, batch 256 and 4,500/85,500/4,500 warmup/constant/decay iterations, with learning rates 7.5e-5 to 4.25e-4 to zero. Mixing both camera views without conditioning degraded performance.
Go to primary source ↓e24PDF pp. 40–41, Figure 16 and Appendix B.4, least-squares and rotation equations
For each camera position, 201 trajectory steps produce 200 inferred/real planar action pairs. A fitted linear map and its rotation component measure action-axis error. The graph shows viewpoint sensitivity; calibration is proposed but explicitly not applied in the reported experiments.
Go to primary source ↓e25PDF pp. 37–40, Appendix B.3 and Figure 15
A separate deterministic frame decoder is trained with pixel MSE for interpretation. Decoded AC rollouts compare open/closed grippers and show cup-position drift. This decoder visualizes features rather than participating in MPC.
Go to primary source ↓e26PDF pp. 33–34, Appendix A.2 and Table 11
Curation discards scenes shorter than four seconds, uses DINOv2 middle-frame embeddings, retrieves target-supported clusters and reweights them by target-dataset membership.
Go to primary source ↓e27PDF p. 7, Section 2.4 Efficient Progressive-Resolution Training; p. 8, Figure 5 caption
The prose gives 12K warmup plus 228K constant plus 12K cooldown; the Figure 5 caption instead describes 252K low-resolution iterations followed by 12K cooldown. Full-resolution compute in that figure is a projection.
Go to primary source ↓e28PDF pp. 46–48, Appendix E.2–E.3 and Tables 21–22
Controlled training reports 128 H100 GPUs and PyTorch 2.5.1/Transformers 4.46.0/Flash Attention 2/DeepSpeed 0.14.4. Scaled Stages 2–3 use 512 H100 GPUs and an unpooled MLP projector. E.3 gives Stage-3 batch 1024, while Table 22 gives 2048.
Go to primary source ↓e30PDF p. 12, Figure 8 caption and Section 4.1 Robot deployment
The source calls the per-action constraint an L1 ball of radius 0.075 and states approximately 13 cm maximum Cartesian displacement. Its exact implemented constraint is not disambiguated here.
Go to primary source ↓8.5 Primary sources
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning ↗
PDF · 26,965 extracted words
Source fingerprint
9cfcfde5fb0d9730637da5b9e7317825c3f3d09e91f3553e22eeba42c74d2226